Skip to content

Commit 4495904

Browse files
mollyheamazonmollyheamazon
committed
parent fb22b91
author Molly He <[email protected]> 1740529460 -0800 committer Molly He <[email protected]> 1744156034 -0700 parent fb22b91 author Molly He <[email protected]> 1740529460 -0800 committer Molly He <[email protected]> 1744155953 -0700 parent fb22b91 author Molly He <[email protected]> 1740529460 -0800 committer Molly He <[email protected]> 1744155703 -0700 parent fb22b91 author Molly He <[email protected]> 1740529460 -0800 committer Molly He <[email protected]> 1744155683 -0700 Test py312 ci support Test py312 ci support first commit test to point to personal stack changed tox.ini Revert "changed tox.ini" This reverts commit 696cb47. Revert "Revert "changed tox.ini"" This reverts commit 5d35bd0. Revert "Revert "Revert "changed tox.ini""" This reverts commit 2ab95fb. Revert "Revert "changed tox.ini"" This reverts commit 5d35bd0. Revert "changed tox.ini" This reverts commit 696cb47. Revert "test to point to personal stack" This reverts commit 49de844. Revert "first commit" This reverts commit 1374184. add pyproject.toml add py312 to ci bump numpy version numpy version change add py312 upgrade pip version add setuptools wheel to tox.ini add pyyaml version constraint, remove py312 from docstring because there is no py312 image yet update pyyaml version constraint update pyyaml version constraint deprecate py38 bump scipy bump tensorflow and tensorboard bump dill bump apache-airflow to ensure dill and greenlet version remove constraint for apache-airflow remove constraint for apache-airflow bump torch version bump torchvision version new tests try changing some tests update model trainer test fix test_pipeline add constraint to apache in tox Fix key error in _send_metrics() (aws#5068) Co-authored-by: pintaoz <[email protected]> Use sagemaker session's s3_resource in download_folder (aws#5064) Co-authored-by: pintaoz <[email protected]> Fix error when there is no session to call _create_model_request() (aws#5062) * Fix error when there is no session to call _create_model_request() * Fix codestyle --------- Co-authored-by: pintaoz <[email protected]> Ensure Model.is_repack() returns a boolean (aws#5060) * Ensure Model.is_repack() returns a boolean * update test --------- Co-authored-by: pintaoz <[email protected]> feat: Make DistributedConfig Extensible (aws#5039) * feat: Make DistributedConfig Extensible * pylint * Include none types when creating config jsons for safer reference * fix: update test to account for changes * format * Add integ test * pylint * prepare release v2.240.0 * update development version to v2.240.1.dev0 * Fix key error in _send_metrics() (aws#5068) Co-authored-by: pintaoz <[email protected]> * fix: Added check for the presence of model package group before creating one (aws#5063) Co-authored-by: Keshav Chandak <[email protected]> * Use sagemaker session's s3_resource in download_folder (aws#5064) Co-authored-by: pintaoz <[email protected]> * remove union * fix merge artifact * Change dir path to distributed_drivers * update paths --------- Co-authored-by: ci <ci> Co-authored-by: pintaoz-aws <[email protected]> Co-authored-by: pintaoz <[email protected]> Co-authored-by: Keshav Chandak <[email protected]> Co-authored-by: Keshav Chandak <[email protected]> Skip tests with deprecated instance type (aws#5077) Co-authored-by: pintaoz <[email protected]> pipeline definition function doc update (aws#5074) Co-authored-by: Rohan Gujarathi <[email protected]> feat: add integ tests for training JumpStart models in private hub (aws#5076) * feat: add integ tests for training JumpStart models in private hub * fixed formatting * remove unused imports * fix unused imports * fix unit test failure and fix bug around versioning * fix formatting * fix unit tests * fix model_uri usage issue * fix some formatting * separate private hub setup code * add try catch block * fix flake8 issue so except clause is not bare * black formatting fix: resolve infinite loop in _find_config on Windows systems (aws#4970) * fix: resolve Windows path handling in _find_config * Replace Path.match("/") with Path.anchor comparison * Fix infinite loop in _studio.py path traversal * test: Add tests for the new root path exploration * Fix formatting style * Fixed line to long * Fix docstyle by running black manually * Fix testcase with \\ when running on non-windows machines * Fix formatting style * cleanup unused import change: update image_uri_configs 03-11-2025 07:18:09 PST Fixing Pytorch training python version in tests (aws#5084) * Fixing Pytorch training python version in tests * Updating Inference test handling remove s3 output location requirement from hub class init (aws#5081) * remove s3 output location requirement from hub class init * fix integ test hub * lint * fix test --------- Co-authored-by: Gokul Anantha Narayanan <[email protected]> fix: Prevent RunContext overlap between test_run tests (aws#5083) Co-authored-by: Gokul Anantha Narayanan <[email protected]> remove py38 from unit testing fix integ test by bumping py38 to py39 for PyTorch change framework_version that supports py39 in integ tests remove py38 from unit testing Update estimator.py
1 parent fb22b91 commit 4495904

File tree

29 files changed

+203
-193
lines changed

29 files changed

+203
-193
lines changed

‎.githooks/pre-push

+2-2
Original file line numberDiff line numberDiff line change
@@ -12,5 +12,5 @@ start_time=`date +%s`
1212
tox -e sphinx,doc8 --parallel all
1313
./ci-scripts/displaytime.sh 'sphinx,doc8' $start_time
1414
start_time=`date +%s`
15-
tox -e py38,py39,py310 --parallel all -- tests/unit
16-
./ci-scripts/displaytime.sh 'py38,py39,py310 unit' $start_time
15+
tox -e py39,py310,py311,py312 --parallel all -- tests/unit
16+
./ci-scripts/displaytime.sh 'py39,py310,py311,py312 unit' $start_time

‎.github/workflows/codebuild-ci-health.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ jobs:
2626
strategy:
2727
fail-fast: false
2828
matrix:
29-
python-version: ["py38", "py39", "py310", "py311"]
29+
python-version: ["py39", "py310", "py311","py312"]
3030
steps:
3131
- name: Configure AWS Credentials
3232
uses: aws-actions/configure-aws-credentials@v4

‎.github/workflows/codebuild-ci.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ jobs:
6363
strategy:
6464
fail-fast: false
6565
matrix:
66-
python-version: ["py38","py39","py310","py311"]
66+
python-version: ["py39","py310","py311","py312"]
6767
steps:
6868
- name: Configure AWS Credentials
6969
uses: aws-actions/configure-aws-credentials@v4

‎doc/requirements.txt

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
sphinx==5.1.1
22
sphinx-rtd-theme==0.5.0
33
docutils==0.15.2
4-
packaging==20.9
4+
packaging>=23.0,<25
55
jinja2==3.1.6
66
schema==0.7.5
77
accelerate>=0.24.1,<=0.27.0

‎pyproject.toml

+5-5
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ name = "sagemaker"
77
dynamic = ["version", "optional-dependencies"]
88
description = "Open source library for training and deploying models on Amazon SageMaker."
99
readme = "README.rst"
10-
requires-python = ">=3.8"
10+
requires-python = ">=3.9"
1111
authors = [
1212
{ name = "Amazon Web Services" },
1313
]
@@ -25,10 +25,10 @@ classifiers = [
2525
"License :: OSI Approved :: Apache Software License",
2626
"Natural Language :: English",
2727
"Programming Language :: Python",
28-
"Programming Language :: Python :: 3.8",
2928
"Programming Language :: Python :: 3.9",
3029
"Programming Language :: Python :: 3.10",
3130
"Programming Language :: Python :: 3.11",
31+
"Programming Language :: Python :: 3.12",
3232
]
3333
dependencies = [
3434
"attrs>=23.1.0,<24",
@@ -39,15 +39,15 @@ dependencies = [
3939
"google-pasta",
4040
"importlib-metadata>=1.4.0,<7.0",
4141
"jsonschema",
42-
"numpy>=1.9.0,<2.0",
42+
"numpy>=1.26.0,<2.0",
4343
"omegaconf>=2.2,<=2.3",
44-
"packaging>=20.0",
44+
"packaging>=23.0,<25",
4545
"pandas",
4646
"pathos",
4747
"platformdirs",
4848
"protobuf>=3.12,<6.0",
4949
"psutil",
50-
"PyYAML~=6.0",
50+
"PyYAML>=6.0.1",
5151
"requests",
5252
"sagemaker-core>=1.0.17,<2.0.0",
5353
"schema",

‎requirements/extras/local_requirements.txt

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
urllib3>=1.26.8,<3.0.0
22
docker>=5.0.2,<8.0.0
3-
PyYAML>=5.4.1,<7
3+
PyYAML>=6.0.1,<7

‎requirements/extras/scipy_requirements.txt

+1-1
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
scipy==1.10.1
1+
scipy==1.11.3

‎requirements/extras/test_requirements.txt

+5-5
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
tox==3.24.5
2-
numpy>=1.24.0
2+
numpy>=1.26.0,<2.0
33
build[virtualenv]==1.2.1
44
flake8==4.0.1
55
pytest==6.2.5
@@ -14,7 +14,7 @@ awslogs==0.14.0
1414
black==24.3.0
1515
stopit==1.1.2
1616
# Update tox.ini to have correct version of airflow constraints file
17-
apache-airflow==2.9.3
17+
apache-airflow==2.10.4
1818
apache-airflow-providers-amazon==7.2.1
1919
attrs>=23.1.0,<24
2020
fabric==2.6.0
@@ -26,13 +26,13 @@ pandas==1.4.4
2626
scikit-learn==1.3.0
2727
cloudpickle==2.2.1
2828
jsonpickle<4.0.0
29-
PyYAML==6.0
29+
PyYAML>=6.0.1
3030
# TODO find workaround
3131
xgboost>=1.6.2,<=1.7.6
3232
pillow>=10.0.1,<=11
3333
opentelemetry-proto==1.27.0
3434
protobuf==4.25.5
35-
tensorboard>=2.9.0,<=2.15.2
35+
tensorboard>=2.16.2,<=2.18.0
3636
transformers==4.48.0
3737
sentencepiece==0.1.99
3838
# https://github.com/triton-inference-server/server/issues/6246
@@ -42,7 +42,7 @@ onnx==1.17.0
4242
nbformat>=5.9,<6
4343
accelerate>=0.24.1,<=0.27.0
4444
schema==0.7.5
45-
tensorflow>=2.9.0,<=2.15.1
45+
tensorflow>=2.16.2,<=2.18.0
4646
mlflow>=2.12.2,<2.13
4747
huggingface_hub==0.26.2
4848
uvicorn>=0.30.1

‎src/sagemaker/serve/utils/conda_in_process.yml

+5-5
Original file line numberDiff line numberDiff line change
@@ -12,15 +12,15 @@ dependencies:
1212
- boto3>=1.34.142,<2.0
1313
- cloudpickle==2.2.1
1414
- google-pasta
15-
- numpy>=1.9.0,<2.0
15+
- numpy>=1.26.0,<2.0
1616
- protobuf>=3.12,<5.0
1717
- smdebug_rulesconfig==1.0.1
1818
- importlib-metadata>=1.4.0,<7.0
19-
- packaging>=20.0
19+
- packaging>=23.0,<25
2020
- pandas
2121
- pathos
2222
- schema
23-
- PyYAML~=6.0
23+
- PyYAML>=6.0.1
2424
- jsonschema
2525
- platformdirs
2626
- tblib>=1.7.0,<4
@@ -43,7 +43,7 @@ dependencies:
4343
- colorama>=0.4.4
4444
- contextlib2>=21.6.0
4545
- decorator>=5.1.1
46-
- dill>=0.3.6
46+
- dill>=0.3.9
4747
- docutils>=0.16
4848
- entrypoints>=0.4
4949
- filelock>=3.11.0
@@ -82,7 +82,7 @@ dependencies:
8282
- python-dateutil>=2.8.2
8383
- pytz>=2023.3
8484
- pytz-deprecation-shim>=0.1.0.post0
85-
- pyyaml>=5.4.1
85+
- pyyaml>=6.0.1
8686
- regex>=2023.3.23
8787
- requests>=2.28.2
8888
- rich>=13.3.4

‎src/sagemaker/serve/utils/in_process_requirements.txt

+2-2
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ cloudpickle==2.2.1
1111
colorama>=0.4.4
1212
contextlib2>=21.6.0
1313
decorator>=5.1.1
14-
dill>=0.3.6
14+
dill>=0.3.9
1515
docutils>=0.16
1616
entrypoints>=0.4
1717
filelock>=3.11.0
@@ -50,7 +50,7 @@ pyrsistent>=0.19.3
5050
python-dateutil>=2.8.2
5151
pytz>=2023.3
5252
pytz-deprecation-shim>=0.1.0.post0
53-
pyyaml>=5.4.1
53+
pyyaml>=6.0.1
5454
regex>=2023.3.23
5555
requests>=2.28.2
5656
rich>=13.3.4

‎tests/data/pipeline/model_step/pytorch_mnist/requirements.txt

+1-1
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
scipy>=1.8.1
1+
scipy>=1.11.3

‎tests/data/remote_function/requirements.txt

+1-1
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
scipy==1.10.1
1+
scipy==1.11.3

‎tests/data/serve_resources/mlflow/pytorch/conda.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ dependencies:
99
- cffi==1.16.0
1010
- cloudpickle==2.2.1
1111
- defusedxml==0.7.1
12-
- dill==0.3.8
12+
- dill==0.3.9
1313
- gmpy2==2.1.2
1414
- numpy==1.26.4
1515
- opt-einsum==3.3.0

‎tests/data/serve_resources/mlflow/pytorch/requirements.txt

+3-3
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,11 @@ astunparse==1.6.3
33
cffi==1.16.0
44
cloudpickle==2.2.1
55
defusedxml==0.7.1
6-
dill==0.3.8
6+
dill==0.3.9
77
gmpy2==2.1.2
8-
numpy==1.24.4
8+
numpy==1.26.4
99
opt-einsum==3.3.0
10-
packaging==21.3
10+
packaging>=23.0,<25
1111
pandas==2.2.1
1212
pyyaml==6.0.1
1313
requests==2.32.2

‎tests/data/serve_resources/mlflow/xgboost/requirements.txt

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
mlflow==2.13.2
22
lz4==4.3.2
3-
numpy==1.24.4
3+
numpy==1.26.4
44
pandas==2.0.3
55
psutil==5.9.8
66
scikit-learn==1.3.2
7-
scipy==1.10.1
7+
scipy==1.11.3
88
xgboost==1.7.1

‎tests/data/workflow/requirements.txt

+1-1
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
scipy==1.10.1
1+
scipy==1.11.3

‎tests/integ/sagemaker/experiments/test_run.py

+18-8
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@
3232
from sagemaker.experiments.trial_component import _TrialComponent
3333
from sagemaker.sklearn import SKLearn
3434
from sagemaker.utils import retry_with_backoff, unique_name_from_base
35-
from tests.integ.sagemaker.experiments.helpers import name, cleanup_exp_resources
35+
from tests.integ.sagemaker.experiments.helpers import name, cleanup_exp_resources, clear_run_context
3636
from sagemaker.experiments.run import (
3737
RUN_NAME_BASE,
3838
DELIMITER,
@@ -55,7 +55,7 @@ def artifact_file_path(tempdir):
5555
metric_name = "Test-Local-Init-Log-Metric"
5656

5757

58-
def test_local_run_with_load(sagemaker_session, artifact_file_path):
58+
def test_local_run_with_load(sagemaker_session, artifact_file_path, clear_run_context):
5959
exp_name = f"My-Local-Exp-{name()}"
6060
with cleanup_exp_resources(exp_names=[exp_name], sagemaker_session=sagemaker_session):
6161
# Run name is not provided, will create a new TC
@@ -86,7 +86,9 @@ def verify_load_run():
8686
retry_with_backoff(verify_load_run, 4)
8787

8888

89-
def test_two_local_run_init_with_same_run_name_and_different_exp_names(sagemaker_session):
89+
def test_two_local_run_init_with_same_run_name_and_different_exp_names(
90+
sagemaker_session, clear_run_context
91+
):
9092
exp_name1 = f"my-two-local-exp1-{name()}"
9193
exp_name2 = f"my-two-local-exp2-{name()}"
9294
run_name = "test-run"
@@ -124,7 +126,9 @@ def test_two_local_run_init_with_same_run_name_and_different_exp_names(sagemaker
124126
("my-test4", "test-run", "run-display-name-test"), # with supplied display name
125127
],
126128
)
127-
def test_run_name_vs_trial_component_name_edge_cases(sagemaker_session, input_names):
129+
def test_run_name_vs_trial_component_name_edge_cases(
130+
sagemaker_session, input_names, clear_run_context
131+
):
128132
exp_name, run_name, run_display_name = input_names
129133
with cleanup_exp_resources(exp_names=[exp_name], sagemaker_session=sagemaker_session):
130134
with Run(
@@ -177,6 +181,7 @@ def test_run_from_local_and_train_job_and_all_exp_cfg_match(
177181
execution_role,
178182
sagemaker_client_config,
179183
sagemaker_metrics_config,
184+
clear_run_context,
180185
):
181186
# Notes:
182187
# 1. The 1st Run created locally and its exp config was auto passed to the job
@@ -277,6 +282,7 @@ def test_run_from_local_and_train_job_and_exp_cfg_not_match(
277282
execution_role,
278283
sagemaker_client_config,
279284
sagemaker_metrics_config,
285+
clear_run_context,
280286
):
281287
# Notes:
282288
# 1. The 1st Run created locally and its exp config was auto passed to the job
@@ -363,6 +369,7 @@ def test_run_from_train_job_only(
363369
execution_role,
364370
sagemaker_client_config,
365371
sagemaker_metrics_config,
372+
clear_run_context,
366373
):
367374
# Notes:
368375
# 1. No Run created locally or specified in experiment config
@@ -413,6 +420,7 @@ def test_run_from_processing_job_and_override_default_exp_config(
413420
execution_role,
414421
sagemaker_client_config,
415422
sagemaker_metrics_config,
423+
clear_run_context,
416424
):
417425
# Notes:
418426
# 1. The 1st Run (run) created locally
@@ -492,6 +500,7 @@ def test_run_from_transform_job(
492500
execution_role,
493501
sagemaker_client_config,
494502
sagemaker_metrics_config,
503+
clear_run_context,
495504
):
496505
# Notes:
497506
# 1. The 1st Run (run) created locally
@@ -573,6 +582,7 @@ def test_load_run_auto_pass_in_exp_config_to_job(
573582
execution_role,
574583
sagemaker_client_config,
575584
sagemaker_metrics_config,
585+
clear_run_context,
576586
):
577587
# Notes:
578588
# 1. In local side, load the Run created previously and invoke a job under the load context
@@ -621,7 +631,7 @@ def test_load_run_auto_pass_in_exp_config_to_job(
621631
)
622632

623633

624-
def test_list(run_obj, sagemaker_session):
634+
def test_list(run_obj, sagemaker_session, clear_run_context):
625635
tc1 = _TrialComponent.create(
626636
trial_component_name=f"non-run-tc1-{name()}",
627637
sagemaker_session=sagemaker_session,
@@ -643,7 +653,7 @@ def test_list(run_obj, sagemaker_session):
643653
assert run_tcs[0].experiment_config == run_obj.experiment_config
644654

645655

646-
def test_list_twice(run_obj, sagemaker_session):
656+
def test_list_twice(run_obj, sagemaker_session, clear_run_context):
647657
tc1 = _TrialComponent.create(
648658
trial_component_name=f"non-run-tc1-{name()}",
649659
sagemaker_session=sagemaker_session,
@@ -720,8 +730,8 @@ def _generate_processor(
720730
)
721731
return FrameworkProcessor(
722732
estimator_cls=PyTorch,
723-
framework_version="1.10",
724-
py_version="py38",
733+
framework_version="1.13.1",
734+
py_version="py39",
725735
instance_count=1,
726736
instance_type="ml.m5.xlarge",
727737
role=execution_role,

‎tests/integ/sagemaker/workflow/test_workflow.py

+4-4
Original file line numberDiff line numberDiff line change
@@ -1122,8 +1122,8 @@ def test_model_registration_with_tuning_model(
11221122
entry_point=entry_point,
11231123
source_dir=base_dir,
11241124
role=role,
1125-
framework_version="1.10",
1126-
py_version="py38",
1125+
framework_version="1.13.1",
1126+
py_version="py39",
11271127
instance_count=instance_count,
11281128
instance_type=instance_type,
11291129
sagemaker_session=pipeline_session,
@@ -1159,8 +1159,8 @@ def test_model_registration_with_tuning_model(
11591159
),
11601160
entry_point=entry_point,
11611161
source_dir=base_dir,
1162-
framework_version="1.10",
1163-
py_version="py38",
1162+
framework_version="1.13.1",
1163+
py_version="py39",
11641164
sagemaker_session=pipeline_session,
11651165
)
11661166
step_model_regis_args = model.register(

‎tests/unit/sagemaker/huggingface/test_llm_utils.py

+2-2
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ def test_huggingface_model_metadata_unauthorized_exception(self, mock_urllib):
6565
"Trying to access a gated/private HuggingFace model without valid credentials. "
6666
"Please provide a HUGGING_FACE_HUB_TOKEN in env_vars"
6767
)
68-
self.assertEquals(expected_error_msg, str(context.exception))
68+
self.assertEqual(expected_error_msg, str(context.exception))
6969

7070
@patch("sagemaker.huggingface.llm_utils.urllib")
7171
def test_huggingface_model_metadata_general_exception(self, mock_urllib):
@@ -76,7 +76,7 @@ def test_huggingface_model_metadata_general_exception(self, mock_urllib):
7676
expected_error_msg = (
7777
f"Did not find model metadata for the following HuggingFace Model ID {MOCK_HF_ID}"
7878
)
79-
self.assertEquals(expected_error_msg, str(context.exception))
79+
self.assertEqual(expected_error_msg, str(context.exception))
8080

8181
@patch("huggingface_hub.snapshot_download")
8282
def test_download_huggingface_model_metadata(self, mock_snapshot_download):

‎tests/unit/sagemaker/jumpstart/constants.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -14360,7 +14360,7 @@
1436014360
"jmespath==1.0.1",
1436114361
"jsonschema==4.17.3",
1436214362
"multiprocess==0.70.14",
14363-
"numpy==1.24.3",
14363+
"numpy==1.26.4",
1436414364
"oscrypto==1.3.0",
1436514365
"packaging==23.1",
1436614366
"pandas==2.0.2",

0 commit comments

Comments
 (0)