Skip to content

feat: local download dir for Model and Estimator classes #3535

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 47 commits into from
Closed

feat: local download dir for Model and Estimator classes #3535

Show file tree
Hide file tree
Changes from 39 commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
c97c467
fix: type hint of PySparkProcessor __init__ (#3297)
NivekNey Dec 2, 2022
de58941
fix: fix PySparkProcessor __init__ params type (#3354)
andre-marcos-perez Dec 2, 2022
41dd330
fix: Allow Py 3.7 for MMS Test Docker env (#3080)
shreyapandit Dec 2, 2022
1e23a3f
refactoring : using with statement (#3286)
maldil Dec 2, 2022
19efadf
Update local_requirements.txt PyYAML version (#3095)
shreyapandit Dec 2, 2022
76f7782
feature: Update TF 2.9 and TF 2.10 inference DLCs (#3465)
arjkesh Dec 2, 2022
fde0738
feature: Added transform with monitoring pipeline step in transformer…
keshav-chandak Dec 2, 2022
7f9f3b0
fix: Fix bug forcing uploaded tar to be named sourcedir (#3412)
claytonparnell Dec 2, 2022
5d59767
feature: Add Code Owners file (#3503)
navinsoni Dec 2, 2022
0f5cf18
prepare release v2.119.0
Dec 3, 2022
f1f0013
update development version to v2.119.1.dev0
Dec 3, 2022
bb4b689
feature: Add DXB region to frameworks by DLC (#3387)
RadhikaB-97 Dec 5, 2022
b68bcd9
fix: support idempotency for framework and spark processors (#3460)
brockwade633 Dec 5, 2022
32969da
feature: Update registries with new region account number mappings. (…
kenny-ezirim Dec 6, 2022
767da0a
feature: Adding support for SageMaker Training Compiler in PyTorch es…
Lokiiiiii Dec 7, 2022
d779d1b
feature: Add Neo image uri config for Pytorch 1.12 (#3507)
HappyAmazonian Dec 7, 2022
83327fb
prepare release v2.120.0
Dec 7, 2022
5bffb04
update development version to v2.120.1.dev0
Dec 7, 2022
b828396
feature: Algorithms Region Expansion OSU/DXB (#3508)
malav-shastri Dec 7, 2022
357f732
fix: Add constraints file for apache-airflow (#3510)
navinsoni Dec 7, 2022
a28d1dd
fix: FrameworkProcessor S3 uploads (#3493)
brockwade633 Dec 8, 2022
11d2475
prepare release v2.121.0
Dec 8, 2022
24171b5
update development version to v2.121.1.dev0
Dec 8, 2022
d5847d5
Fix: Differentiate SageMaker Training Compiler's PT DLCs from base PT…
Lokiiiiii Dec 8, 2022
3f6ea88
fix: Fix failing jumpstart cache unit tests (#3514)
evakravi Dec 8, 2022
4570aa6
fix: Pop out ModelPackageName from pipeline definition (#3472)
qidewenwhen Dec 9, 2022
959ea1a
prepare release v2.121.1
Dec 9, 2022
b2e8b66
update development version to v2.121.2.dev0
Dec 9, 2022
355975d
fix: Skip Bad Transform Test (#3521)
amzn-choeric Dec 9, 2022
fadc817
fix: Revert "fix: type hint of PySparkProcessor __init__" (#3524)
mufaddal-rohawala Dec 9, 2022
c5fc93f
change: Update for Tensorflow Serving 2.11 inference DLCs (#3509)
hballuru Dec 9, 2022
ec8da98
prepare release v2.121.2
Dec 12, 2022
0352122
update development version to v2.121.3.dev0
Dec 12, 2022
d6c0214
feature: Add OSU region to frameworks for DLC (#3532)
kace Dec 12, 2022
5af4feb
fix: Remove content type image/jpg from analysis configuration schema…
xgchena Dec 12, 2022
4389847
fix: unpin packaging version (#3533)
claytonparnell Dec 13, 2022
ef0c3e0
feat: local download dir for Model and Estimator classes
evakravi Dec 13, 2022
27cfe6c
Merge branch 'master' into feat/local-download-dir
evakravi Dec 13, 2022
f3e0feb
chore: move local_download_dir to SessionSettings
evakravi Dec 13, 2022
d314b67
fix: tox errors
evakravi Dec 14, 2022
f912edf
Merge remote-tracking branch 'origin' into feat/local-download-dir
evakravi Dec 30, 2022
51899a9
chore: clean git diff
evakravi Dec 30, 2022
eb3ba1b
Merge remote-tracking branch 'origin' into feat/local-download-dir
evakravi Jan 5, 2023
094e476
fix: pytorch compiler unit tests
evakravi Jan 5, 2023
0086f6b
Merge branch 'master' into feat/local-download-dir
evakravi Jan 12, 2023
7ff0891
Merge branch 'master' into feat/local-download-dir
evakravi Jan 17, 2023
b6f9918
Merge branch 'master' into feat/local-download-dir
evakravi Jan 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,72 @@
# Changelog

## v2.121.2 (2022-12-12)

### Bug Fixes and Other Changes

* Update for Tensorflow Serving 2.11 inference DLCs
* Revert "fix: type hint of PySparkProcessor __init__"
* Skip Bad Transform Test

## v2.121.1 (2022-12-09)

### Bug Fixes and Other Changes

* Pop out ModelPackageName from pipeline definition
* Fix failing jumpstart cache unit tests

## v2.121.0 (2022-12-08)

### Features

* Algorithms Region Expansion OSU/DXB

### Bug Fixes and Other Changes

* FrameworkProcessor S3 uploads
* Add constraints file for apache-airflow

## v2.120.0 (2022-12-07)

### Features

* Add Neo image uri config for Pytorch 1.12
* Adding support for SageMaker Training Compiler in PyTorch estimator starting 1.12
* Update registries with new region account number mappings.
* Add DXB region to frameworks by DLC

### Bug Fixes and Other Changes

* support idempotency for framework and spark processors

## v2.119.0 (2022-12-03)

### Features

* Add Code Owners file
* Added transform with monitoring pipeline step in transformer
* Update TF 2.9 and TF 2.10 inference DLCs
* make estimator accept json file as modelparallel config
* SageMaker Training Compiler does not support p4de instances
* Add support for SparkML v3.3

### Bug Fixes and Other Changes

* Fix bug forcing uploaded tar to be named sourcedir
* Update local_requirements.txt PyYAML version
* refactoring : using with statement
* Allow Py 3.7 for MMS Test Docker env
* fix PySparkProcessor __init__ params type
* type hint of PySparkProcessor __init__
* Return ARM XGB/SKLearn tags if `image_scope` is `inference_graviton`
* Update scipy to 1.7.3 to support M1 development envs
* Fixing type hints for Spark processor that has instance type/count params in reverse order
* Add DeepAR ap-northeast-3 repository.
* Fix AsyncInferenceConfig documentation typo
* fix ml_inf to ml_inf1 in Neo multi-version support
* Fix type annotations
* add neo mvp region accounts

## v2.118.0 (2022-12-01)

### Features
Expand Down
1 change: 1 addition & 0 deletions CODEOWNERS
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* @aws/sagemaker-ml-frameworks
2 changes: 1 addition & 1 deletion VERSION
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2.118.1.dev0
2.121.3.dev0
1 change: 1 addition & 0 deletions requirements/extras/test_requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ contextlib2==21.6.0
awslogs==0.14.0
black==22.3.0
stopit==1.1.2
# Update tox.ini to have correct version of airflow constraints file
apache-airflow==2.4.1
apache-airflow-providers-amazon==4.0.0
attrs==22.1.0
Expand Down
1 change: 0 additions & 1 deletion src/sagemaker/clarify.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -282,7 +282,6 @@
"text/csv",
"application/jsonlines",
"image/jpeg",
"image/jpg",
"image/png",
"application/x-npy",
),
Expand Down
16 changes: 14 additions & 2 deletions src/sagemaker/fw_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -418,7 +418,19 @@ def tar_and_upload_dir(
script_name = script if directory else os.path.basename(script)
dependencies = dependencies or []
key = "%s/sourcedir.tar.gz" % s3_key_prefix
tmp = tempfile.mkdtemp()
if (
settings is not None
and settings.local_download_dir is not None
and not (
os.path.exists(settings.local_download_dir)
and os.path.isdir(settings.local_download_dir)
)
):
raise ValueError(
f"Inputted directory for storing newly generated temporary directory does not exist: '{settings.local_download_dir}'"
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about attempting to create the target dir with os.makedirs(dir, exists_ok=True) instead?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather not. Making directories on behalf of customers seems risky.

local_download_dir = None if settings == None else settings.local_download_dir
tmp = tempfile.mkdtemp(dir=local_download_dir)
encrypt_artifact = True if settings is None else settings.encrypt_repacked_artifacts

try:
Expand Down Expand Up @@ -493,7 +505,7 @@ def framework_name_from_image(image_uri):
# We must support both the legacy and current image name format.
name_pattern = re.compile(
r"""^(?:sagemaker(?:-rl)?-)?
(tensorflow|mxnet|chainer|pytorch|scikit-learn|xgboost
(tensorflow|mxnet|chainer|pytorch|pytorch-trcomp|scikit-learn|xgboost
|huggingface-tensorflow|huggingface-pytorch
|huggingface-tensorflow-trcomp|huggingface-pytorch-trcomp)(?:-)?
(scriptmode|training)?
Expand Down
5 changes: 2 additions & 3 deletions src/sagemaker/git_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -279,9 +279,8 @@ def _run_clone_command(repo_url, dest_dir):
subprocess.check_call(["git", "clone", repo_url, dest_dir], env=my_env)
elif repo_url.startswith("git@"):
with tempfile.NamedTemporaryFile() as sshnoprompt:
write_pipe = open(sshnoprompt.name, "w")
write_pipe.write("ssh -oBatchMode=yes $@")
write_pipe.close()
with open(sshnoprompt.name, "w") as write_pipe:
write_pipe.write("ssh -oBatchMode=yes $@")
os.chmod(sshnoprompt.name, 0o511)
my_env["GIT_SSH"] = sshnoprompt.name
subprocess.check_call(["git", "clone", repo_url, dest_dir], env=my_env)
Expand Down
42 changes: 42 additions & 0 deletions src/sagemaker/image_uri_config/autogluon.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,11 @@
"eu-west-3": "763104351884",
"eu-south-1": "692866216735",
"me-south-1": "217643126080",
"me-central-1": "914824155844",
"sa-east-1": "763104351884",
"us-east-1": "763104351884",
"us-east-2": "763104351884",
"us-gov-east-1": "446045086412",
"us-gov-west-1": "442386744353",
"us-iso-east-1": "886529160074",
"us-west-1": "763104351884",
Expand Down Expand Up @@ -56,9 +58,11 @@
"eu-west-3": "763104351884",
"eu-south-1": "692866216735",
"me-south-1": "217643126080",
"me-central-1": "914824155844",
"sa-east-1": "763104351884",
"us-east-1": "763104351884",
"us-east-2": "763104351884",
"us-gov-east-1": "446045086412",
"us-gov-west-1": "442386744353",
"us-iso-east-1": "886529160074",
"us-west-1": "763104351884",
Expand Down Expand Up @@ -86,9 +90,11 @@
"eu-west-3": "763104351884",
"eu-south-1": "692866216735",
"me-south-1": "217643126080",
"me-central-1": "914824155844",
"sa-east-1": "763104351884",
"us-east-1": "763104351884",
"us-east-2": "763104351884",
"us-gov-east-1": "446045086412",
"us-gov-west-1": "442386744353",
"us-iso-east-1": "886529160074",
"us-west-1": "763104351884",
Expand Down Expand Up @@ -116,9 +122,11 @@
"eu-west-3": "763104351884",
"eu-south-1": "692866216735",
"me-south-1": "217643126080",
"me-central-1": "914824155844",
"sa-east-1": "763104351884",
"us-east-1": "763104351884",
"us-east-2": "763104351884",
"us-gov-east-1": "446045086412",
"us-gov-west-1": "442386744353",
"us-iso-east-1": "886529160074",
"us-west-1": "763104351884",
Expand Down Expand Up @@ -146,9 +154,11 @@
"eu-west-3": "763104351884",
"eu-south-1": "692866216735",
"me-south-1": "217643126080",
"me-central-1": "914824155844",
"sa-east-1": "763104351884",
"us-east-1": "763104351884",
"us-east-2": "763104351884",
"us-gov-east-1": "446045086412",
"us-gov-west-1": "442386744353",
"us-iso-east-1": "886529160074",
"us-west-1": "763104351884",
Expand Down Expand Up @@ -176,9 +186,11 @@
"eu-west-3": "763104351884",
"eu-south-1": "692866216735",
"me-south-1": "217643126080",
"me-central-1": "914824155844",
"sa-east-1": "763104351884",
"us-east-1": "763104351884",
"us-east-2": "763104351884",
"us-gov-east-1": "446045086412",
"us-gov-west-1": "442386744353",
"us-iso-east-1": "886529160074",
"us-west-1": "763104351884",
Expand All @@ -204,22 +216,27 @@
"ap-northeast-2": "763104351884",
"ap-northeast-3": "364406365360",
"ap-south-1": "763104351884",
"ap-south-2": "772153158452",
"ap-southeast-1": "763104351884",
"ap-southeast-2": "763104351884",
"ap-southeast-3": "907027046896",
"ca-central-1": "763104351884",
"cn-north-1": "727897471807",
"cn-northwest-1": "727897471807",
"eu-central-1": "763104351884",
"eu-central-2": "380420809688",
"eu-north-1": "763104351884",
"eu-west-1": "763104351884",
"eu-west-2": "763104351884",
"eu-west-3": "763104351884",
"eu-south-1": "692866216735",
"eu-south-2": "503227376785",
"me-south-1": "217643126080",
"me-central-1": "914824155844",
"sa-east-1": "763104351884",
"us-east-1": "763104351884",
"us-east-2": "763104351884",
"us-gov-east-1": "446045086412",
"us-gov-west-1": "442386744353",
"us-iso-east-1": "886529160074",
"us-west-1": "763104351884",
Expand All @@ -237,22 +254,27 @@
"ap-northeast-2": "763104351884",
"ap-northeast-3": "364406365360",
"ap-south-1": "763104351884",
"ap-south-2": "772153158452",
"ap-southeast-1": "763104351884",
"ap-southeast-2": "763104351884",
"ap-southeast-3": "907027046896",
"ca-central-1": "763104351884",
"cn-north-1": "727897471807",
"cn-northwest-1": "727897471807",
"eu-central-1": "763104351884",
"eu-central-2": "380420809688",
"eu-north-1": "763104351884",
"eu-west-1": "763104351884",
"eu-west-2": "763104351884",
"eu-west-3": "763104351884",
"eu-south-1": "692866216735",
"eu-south-2": "503227376785",
"me-south-1": "217643126080",
"me-central-1": "914824155844",
"sa-east-1": "763104351884",
"us-east-1": "763104351884",
"us-east-2": "763104351884",
"us-gov-east-1": "446045086412",
"us-gov-west-1": "442386744353",
"us-iso-east-1": "886529160074",
"us-west-1": "763104351884",
Expand All @@ -270,22 +292,27 @@
"ap-northeast-2": "763104351884",
"ap-northeast-3": "364406365360",
"ap-south-1": "763104351884",
"ap-south-2": "772153158452",
"ap-southeast-1": "763104351884",
"ap-southeast-2": "763104351884",
"ap-southeast-3": "907027046896",
"ca-central-1": "763104351884",
"cn-north-1": "727897471807",
"cn-northwest-1": "727897471807",
"eu-central-1": "763104351884",
"eu-central-2": "380420809688",
"eu-north-1": "763104351884",
"eu-west-1": "763104351884",
"eu-west-2": "763104351884",
"eu-west-3": "763104351884",
"eu-south-1": "692866216735",
"eu-south-2": "503227376785",
"me-south-1": "217643126080",
"me-central-1": "914824155844",
"sa-east-1": "763104351884",
"us-east-1": "763104351884",
"us-east-2": "763104351884",
"us-gov-east-1": "446045086412",
"us-gov-west-1": "442386744353",
"us-iso-east-1": "886529160074",
"us-west-1": "763104351884",
Expand All @@ -303,22 +330,27 @@
"ap-northeast-2": "763104351884",
"ap-northeast-3": "364406365360",
"ap-south-1": "763104351884",
"ap-south-2": "772153158452",
"ap-southeast-1": "763104351884",
"ap-southeast-2": "763104351884",
"ap-southeast-3": "907027046896",
"ca-central-1": "763104351884",
"cn-north-1": "727897471807",
"cn-northwest-1": "727897471807",
"eu-central-1": "763104351884",
"eu-central-2": "380420809688",
"eu-north-1": "763104351884",
"eu-west-1": "763104351884",
"eu-west-2": "763104351884",
"eu-west-3": "763104351884",
"eu-south-1": "692866216735",
"eu-south-2": "503227376785",
"me-south-1": "217643126080",
"me-central-1": "914824155844",
"sa-east-1": "763104351884",
"us-east-1": "763104351884",
"us-east-2": "763104351884",
"us-gov-east-1": "446045086412",
"us-gov-west-1": "442386744353",
"us-iso-east-1": "886529160074",
"us-west-1": "763104351884",
Expand All @@ -336,22 +368,27 @@
"ap-northeast-2": "763104351884",
"ap-northeast-3": "364406365360",
"ap-south-1": "763104351884",
"ap-south-2": "772153158452",
"ap-southeast-1": "763104351884",
"ap-southeast-2": "763104351884",
"ap-southeast-3": "907027046896",
"ca-central-1": "763104351884",
"cn-north-1": "727897471807",
"cn-northwest-1": "727897471807",
"eu-central-1": "763104351884",
"eu-central-2": "380420809688",
"eu-north-1": "763104351884",
"eu-west-1": "763104351884",
"eu-west-2": "763104351884",
"eu-west-3": "763104351884",
"eu-south-1": "692866216735",
"eu-south-2": "503227376785",
"me-south-1": "217643126080",
"me-central-1": "914824155844",
"sa-east-1": "763104351884",
"us-east-1": "763104351884",
"us-east-2": "763104351884",
"us-gov-east-1": "446045086412",
"us-gov-west-1": "442386744353",
"us-iso-east-1": "886529160074",
"us-west-1": "763104351884",
Expand All @@ -369,22 +406,27 @@
"ap-northeast-2": "763104351884",
"ap-northeast-3": "364406365360",
"ap-south-1": "763104351884",
"ap-south-2": "772153158452",
"ap-southeast-1": "763104351884",
"ap-southeast-2": "763104351884",
"ap-southeast-3": "907027046896",
"ca-central-1": "763104351884",
"cn-north-1": "727897471807",
"cn-northwest-1": "727897471807",
"eu-central-1": "763104351884",
"eu-central-2": "380420809688",
"eu-north-1": "763104351884",
"eu-west-1": "763104351884",
"eu-west-2": "763104351884",
"eu-west-3": "763104351884",
"eu-south-1": "692866216735",
"eu-south-2": "503227376785",
"me-south-1": "217643126080",
"me-central-1": "914824155844",
"sa-east-1": "763104351884",
"us-east-1": "763104351884",
"us-east-2": "763104351884",
"us-gov-east-1": "446045086412",
"us-gov-west-1": "442386744353",
"us-iso-east-1": "886529160074",
"us-west-1": "763104351884",
Expand Down
2 changes: 2 additions & 0 deletions src/sagemaker/image_uri_config/blazingtext.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,12 @@
"eu-west-3": "749696950732",
"eu-south-1": "257386234256",
"me-south-1": "249704162688",
"me-central-1": "272398656194",
"sa-east-1": "855470959533",
"us-east-1": "811284229777",
"us-east-2": "825641698319",
"us-gov-west-1": "226302683700",
"us-gov-east-1": "237065988967",
"us-iso-east-1": "490574956308",
"us-west-1": "632365934929",
"us-west-2": "433757028032"
Expand Down
Loading