Skip to content

Commit e283a1a

Browse files
wcarpenter1-godaddywcarpenter1-godaddy
committed
Merge branch 'user-short-lived-credentials-override' of github.com:wcarpenter1-godaddy/sagemaker-python-sdk into user-short-lived-credentials-override
2 parents 07a92f2 + 28dbc5d commit e283a1a

File tree

93 files changed

+3612
-230
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

93 files changed

+3612
-230
lines changed

‎CHANGELOG.md

+52
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,57 @@
11
# Changelog
22

3+
## v2.121.0 (2022-12-08)
4+
5+
### Features
6+
7+
* Algorithms Region Expansion OSU/DXB
8+
9+
### Bug Fixes and Other Changes
10+
11+
* FrameworkProcessor S3 uploads
12+
* Add constraints file for apache-airflow
13+
14+
## v2.120.0 (2022-12-07)
15+
16+
### Features
17+
18+
* Add Neo image uri config for Pytorch 1.12
19+
* Adding support for SageMaker Training Compiler in PyTorch estimator starting 1.12
20+
* Update registries with new region account number mappings.
21+
* Add DXB region to frameworks by DLC
22+
23+
### Bug Fixes and Other Changes
24+
25+
* support idempotency for framework and spark processors
26+
27+
## v2.119.0 (2022-12-03)
28+
29+
### Features
30+
31+
* Add Code Owners file
32+
* Added transform with monitoring pipeline step in transformer
33+
* Update TF 2.9 and TF 2.10 inference DLCs
34+
* make estimator accept json file as modelparallel config
35+
* SageMaker Training Compiler does not support p4de instances
36+
* Add support for SparkML v3.3
37+
38+
### Bug Fixes and Other Changes
39+
40+
* Fix bug forcing uploaded tar to be named sourcedir
41+
* Update local_requirements.txt PyYAML version
42+
* refactoring : using with statement
43+
* Allow Py 3.7 for MMS Test Docker env
44+
* fix PySparkProcessor __init__ params type
45+
* type hint of PySparkProcessor __init__
46+
* Return ARM XGB/SKLearn tags if `image_scope` is `inference_graviton`
47+
* Update scipy to 1.7.3 to support M1 development envs
48+
* Fixing type hints for Spark processor that has instance type/count params in reverse order
49+
* Add DeepAR ap-northeast-3 repository.
50+
* Fix AsyncInferenceConfig documentation typo
51+
* fix ml_inf to ml_inf1 in Neo multi-version support
52+
* Fix type annotations
53+
* add neo mvp region accounts
54+
355
## v2.118.0 (2022-12-01)
456

557
### Features

‎CODEOWNERS

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
* @aws/sagemaker-ml-frameworks

‎README.rst

+1-1
Original file line numberDiff line numberDiff line change
@@ -214,7 +214,7 @@ In order to host a SparkML model in SageMaker, it should be serialized with ``ML
214214

215215
For more information on MLeap, see https://github.com/combust/mleap .
216216

217-
Supported major version of Spark: 2.4 (MLeap version - 0.9.6)
217+
Supported major version of Spark: 3.3 (MLeap version - 0.20.0)
218218

219219
Here is an example on how to create an instance of ``SparkMLModel`` class and use ``deploy()`` method to create an
220220
endpoint which can be used to perform prediction against your trained SparkML Model.

‎VERSION

+1-1
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
2.118.1.dev0
1+
2.121.1.dev0

‎doc/overview.rst

+2-2
Original file line numberDiff line numberDiff line change
@@ -1171,7 +1171,7 @@ be ``s3://sagemaker-{REGION}-{ACCOUNTID}/async-endpoint-outputs/{UNIQUE-JOB-NAME
11711171
from sagemaker.async_inference import AsyncInferenceConfig
11721172
11731173
# Create an empty AsyncInferenceConfig object to use default values
1174-
async_config = new AsyncInferenceConfig()
1174+
async_config = AsyncInferenceConfig()
11751175
11761176
Or you can specify configurations in ``AsyncInferenceConfig`` as you like. All of those configuration parameters
11771177
are optional but if you don’t specify the ``output_path``, Amazon SageMaker will use the default ``S3OutputPath``
@@ -1180,7 +1180,7 @@ mentioned above (example shown below):
11801180
.. code:: python
11811181
11821182
# Specify S3OutputPath, MaxConcurrentInvocationsPerInstance and NotificationConfig in the async config object
1183-
async_config = new AsyncInferenceConfig(
1183+
async_config = AsyncInferenceConfig(
11841184
output_path="s3://{s3_bucket}/{bucket_prefix}/output",
11851185
max_concurrent_invocations_per_instance=10,
11861186
notification_config = {

‎requirements/extras/scipy_requirements.txt

+1-1
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
scipy==1.7.2
1+
scipy==1.7.3

‎requirements/extras/test_requirements.txt

+1
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ contextlib2==21.6.0
1111
awslogs==0.14.0
1212
black==22.3.0
1313
stopit==1.1.2
14+
# Update tox.ini to have correct version of airflow constraints file
1415
apache-airflow==2.4.1
1516
apache-airflow-providers-amazon==4.0.0
1617
attrs==22.1.0

‎src/sagemaker/fw_utils.py

+38-1
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
"""Utility methods used by framework classes"""
1414
from __future__ import absolute_import
1515

16+
import json
1617
import logging
1718
import os
1819
import re
@@ -234,6 +235,41 @@ def validate_source_code_input_against_pipeline_variables(
234235
)
235236

236237

238+
def parse_mp_parameters(params):
239+
"""Parse the model parallelism parameters provided by the user.
240+
241+
Args:
242+
params: a string representing path to an existing config, or
243+
a config dict.
244+
245+
Returns:
246+
parsed: a dict of parsed config.
247+
248+
Raises:
249+
ValueError: if params is not a string or a dict, or
250+
the config file cannot be parsed as json.
251+
"""
252+
parsed = None
253+
if isinstance(params, dict):
254+
parsed = params
255+
elif os.path.exists(params):
256+
try:
257+
with open(params, "r") as fp:
258+
parsed = json.load(fp)
259+
except json.decoder.JSONDecodeError:
260+
pass
261+
else:
262+
raise ValueError(
263+
f"Expected a string path to an existing modelparallel config, or a dictionary. "
264+
f"Received: {params}."
265+
)
266+
267+
if parsed is None:
268+
raise ValueError(f"Cannot parse {params} as a json file.")
269+
270+
return parsed
271+
272+
237273
def get_mp_parameters(distribution):
238274
"""Get the model parallelism parameters provided by the user.
239275
@@ -250,6 +286,7 @@ def get_mp_parameters(distribution):
250286
mp_dict = {}
251287
if mp_dict.get("enabled", False) is True:
252288
params = mp_dict.get("parameters", {})
289+
params = parse_mp_parameters(params)
253290
validate_mp_config(params)
254291
return params
255292
return None
@@ -456,7 +493,7 @@ def framework_name_from_image(image_uri):
456493
# We must support both the legacy and current image name format.
457494
name_pattern = re.compile(
458495
r"""^(?:sagemaker(?:-rl)?-)?
459-
(tensorflow|mxnet|chainer|pytorch|scikit-learn|xgboost
496+
(tensorflow|mxnet|chainer|pytorch|pytorch-trcomp|scikit-learn|xgboost
460497
|huggingface-tensorflow|huggingface-pytorch
461498
|huggingface-tensorflow-trcomp|huggingface-pytorch-trcomp)(?:-)?
462499
(scriptmode|training)?

‎src/sagemaker/git_utils.py

+2-3
Original file line numberDiff line numberDiff line change
@@ -279,9 +279,8 @@ def _run_clone_command(repo_url, dest_dir):
279279
subprocess.check_call(["git", "clone", repo_url, dest_dir], env=my_env)
280280
elif repo_url.startswith("git@"):
281281
with tempfile.NamedTemporaryFile() as sshnoprompt:
282-
write_pipe = open(sshnoprompt.name, "w")
283-
write_pipe.write("ssh -oBatchMode=yes $@")
284-
write_pipe.close()
282+
with open(sshnoprompt.name, "w") as write_pipe:
283+
write_pipe.write("ssh -oBatchMode=yes $@")
285284
os.chmod(sshnoprompt.name, 0o511)
286285
my_env["GIT_SSH"] = sshnoprompt.name
287286
subprocess.check_call(["git", "clone", repo_url, dest_dir], env=my_env)

‎src/sagemaker/huggingface/training_compiler/config.py

+2-5
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@
2626
class TrainingCompilerConfig(BaseConfig):
2727
"""The SageMaker Training Compiler configuration class."""
2828

29-
SUPPORTED_INSTANCE_CLASS_PREFIXES = ["p3", "g4dn", "p4d", "g5"]
29+
SUPPORTED_INSTANCE_CLASS_PREFIXES = ["p3", "p3dn", "g4dn", "p4d", "g5"]
3030
SUPPORTED_INSTANCE_TYPES_WITH_EFA = [
3131
"ml.g4dn.8xlarge",
3232
"ml.g4dn.12xlarge",
@@ -87,10 +87,7 @@ def __init__(
8787
super(TrainingCompilerConfig, self).__init__(enabled=enabled, debug=debug)
8888

8989
@classmethod
90-
def validate(
91-
cls,
92-
estimator,
93-
):
90+
def validate(cls, estimator):
9491
"""Checks if SageMaker Training Compiler is configured correctly.
9592
9693
Args:

‎src/sagemaker/image_uri_config/autogluon.json

+30
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
"eu-west-3": "763104351884",
2727
"eu-south-1": "692866216735",
2828
"me-south-1": "217643126080",
29+
"me-central-1": "914824155844",
2930
"sa-east-1": "763104351884",
3031
"us-east-1": "763104351884",
3132
"us-east-2": "763104351884",
@@ -56,6 +57,7 @@
5657
"eu-west-3": "763104351884",
5758
"eu-south-1": "692866216735",
5859
"me-south-1": "217643126080",
60+
"me-central-1": "914824155844",
5961
"sa-east-1": "763104351884",
6062
"us-east-1": "763104351884",
6163
"us-east-2": "763104351884",
@@ -86,6 +88,7 @@
8688
"eu-west-3": "763104351884",
8789
"eu-south-1": "692866216735",
8890
"me-south-1": "217643126080",
91+
"me-central-1": "914824155844",
8992
"sa-east-1": "763104351884",
9093
"us-east-1": "763104351884",
9194
"us-east-2": "763104351884",
@@ -116,6 +119,7 @@
116119
"eu-west-3": "763104351884",
117120
"eu-south-1": "692866216735",
118121
"me-south-1": "217643126080",
122+
"me-central-1": "914824155844",
119123
"sa-east-1": "763104351884",
120124
"us-east-1": "763104351884",
121125
"us-east-2": "763104351884",
@@ -146,6 +150,7 @@
146150
"eu-west-3": "763104351884",
147151
"eu-south-1": "692866216735",
148152
"me-south-1": "217643126080",
153+
"me-central-1": "914824155844",
149154
"sa-east-1": "763104351884",
150155
"us-east-1": "763104351884",
151156
"us-east-2": "763104351884",
@@ -176,6 +181,7 @@
176181
"eu-west-3": "763104351884",
177182
"eu-south-1": "692866216735",
178183
"me-south-1": "217643126080",
184+
"me-central-1": "914824155844",
179185
"sa-east-1": "763104351884",
180186
"us-east-1": "763104351884",
181187
"us-east-2": "763104351884",
@@ -204,19 +210,23 @@
204210
"ap-northeast-2": "763104351884",
205211
"ap-northeast-3": "364406365360",
206212
"ap-south-1": "763104351884",
213+
"ap-south-2": "772153158452",
207214
"ap-southeast-1": "763104351884",
208215
"ap-southeast-2": "763104351884",
209216
"ap-southeast-3": "907027046896",
210217
"ca-central-1": "763104351884",
211218
"cn-north-1": "727897471807",
212219
"cn-northwest-1": "727897471807",
213220
"eu-central-1": "763104351884",
221+
"eu-central-2": "380420809688",
214222
"eu-north-1": "763104351884",
215223
"eu-west-1": "763104351884",
216224
"eu-west-2": "763104351884",
217225
"eu-west-3": "763104351884",
218226
"eu-south-1": "692866216735",
227+
"eu-south-2": "503227376785",
219228
"me-south-1": "217643126080",
229+
"me-central-1": "914824155844",
220230
"sa-east-1": "763104351884",
221231
"us-east-1": "763104351884",
222232
"us-east-2": "763104351884",
@@ -237,19 +247,23 @@
237247
"ap-northeast-2": "763104351884",
238248
"ap-northeast-3": "364406365360",
239249
"ap-south-1": "763104351884",
250+
"ap-south-2": "772153158452",
240251
"ap-southeast-1": "763104351884",
241252
"ap-southeast-2": "763104351884",
242253
"ap-southeast-3": "907027046896",
243254
"ca-central-1": "763104351884",
244255
"cn-north-1": "727897471807",
245256
"cn-northwest-1": "727897471807",
246257
"eu-central-1": "763104351884",
258+
"eu-central-2": "380420809688",
247259
"eu-north-1": "763104351884",
248260
"eu-west-1": "763104351884",
249261
"eu-west-2": "763104351884",
250262
"eu-west-3": "763104351884",
251263
"eu-south-1": "692866216735",
264+
"eu-south-2": "503227376785",
252265
"me-south-1": "217643126080",
266+
"me-central-1": "914824155844",
253267
"sa-east-1": "763104351884",
254268
"us-east-1": "763104351884",
255269
"us-east-2": "763104351884",
@@ -270,19 +284,23 @@
270284
"ap-northeast-2": "763104351884",
271285
"ap-northeast-3": "364406365360",
272286
"ap-south-1": "763104351884",
287+
"ap-south-2": "772153158452",
273288
"ap-southeast-1": "763104351884",
274289
"ap-southeast-2": "763104351884",
275290
"ap-southeast-3": "907027046896",
276291
"ca-central-1": "763104351884",
277292
"cn-north-1": "727897471807",
278293
"cn-northwest-1": "727897471807",
279294
"eu-central-1": "763104351884",
295+
"eu-central-2": "380420809688",
280296
"eu-north-1": "763104351884",
281297
"eu-west-1": "763104351884",
282298
"eu-west-2": "763104351884",
283299
"eu-west-3": "763104351884",
284300
"eu-south-1": "692866216735",
301+
"eu-south-2": "503227376785",
285302
"me-south-1": "217643126080",
303+
"me-central-1": "914824155844",
286304
"sa-east-1": "763104351884",
287305
"us-east-1": "763104351884",
288306
"us-east-2": "763104351884",
@@ -303,19 +321,23 @@
303321
"ap-northeast-2": "763104351884",
304322
"ap-northeast-3": "364406365360",
305323
"ap-south-1": "763104351884",
324+
"ap-south-2": "772153158452",
306325
"ap-southeast-1": "763104351884",
307326
"ap-southeast-2": "763104351884",
308327
"ap-southeast-3": "907027046896",
309328
"ca-central-1": "763104351884",
310329
"cn-north-1": "727897471807",
311330
"cn-northwest-1": "727897471807",
312331
"eu-central-1": "763104351884",
332+
"eu-central-2": "380420809688",
313333
"eu-north-1": "763104351884",
314334
"eu-west-1": "763104351884",
315335
"eu-west-2": "763104351884",
316336
"eu-west-3": "763104351884",
317337
"eu-south-1": "692866216735",
338+
"eu-south-2": "503227376785",
318339
"me-south-1": "217643126080",
340+
"me-central-1": "914824155844",
319341
"sa-east-1": "763104351884",
320342
"us-east-1": "763104351884",
321343
"us-east-2": "763104351884",
@@ -336,19 +358,23 @@
336358
"ap-northeast-2": "763104351884",
337359
"ap-northeast-3": "364406365360",
338360
"ap-south-1": "763104351884",
361+
"ap-south-2": "772153158452",
339362
"ap-southeast-1": "763104351884",
340363
"ap-southeast-2": "763104351884",
341364
"ap-southeast-3": "907027046896",
342365
"ca-central-1": "763104351884",
343366
"cn-north-1": "727897471807",
344367
"cn-northwest-1": "727897471807",
345368
"eu-central-1": "763104351884",
369+
"eu-central-2": "380420809688",
346370
"eu-north-1": "763104351884",
347371
"eu-west-1": "763104351884",
348372
"eu-west-2": "763104351884",
349373
"eu-west-3": "763104351884",
350374
"eu-south-1": "692866216735",
375+
"eu-south-2": "503227376785",
351376
"me-south-1": "217643126080",
377+
"me-central-1": "914824155844",
352378
"sa-east-1": "763104351884",
353379
"us-east-1": "763104351884",
354380
"us-east-2": "763104351884",
@@ -369,19 +395,23 @@
369395
"ap-northeast-2": "763104351884",
370396
"ap-northeast-3": "364406365360",
371397
"ap-south-1": "763104351884",
398+
"ap-south-2": "772153158452",
372399
"ap-southeast-1": "763104351884",
373400
"ap-southeast-2": "763104351884",
374401
"ap-southeast-3": "907027046896",
375402
"ca-central-1": "763104351884",
376403
"cn-north-1": "727897471807",
377404
"cn-northwest-1": "727897471807",
378405
"eu-central-1": "763104351884",
406+
"eu-central-2": "380420809688",
379407
"eu-north-1": "763104351884",
380408
"eu-west-1": "763104351884",
381409
"eu-west-2": "763104351884",
382410
"eu-west-3": "763104351884",
383411
"eu-south-1": "692866216735",
412+
"eu-south-2": "503227376785",
384413
"me-south-1": "217643126080",
414+
"me-central-1": "914824155844",
385415
"sa-east-1": "763104351884",
386416
"us-east-1": "763104351884",
387417
"us-east-2": "763104351884",

0 commit comments

Comments
 (0)