Skip to content

Commit 08a31b4

Browse files
author
Dan
authored
Merge branch 'master' into mask-creds-local-mode
2 parents 427ece5 + 6acde94 commit 08a31b4

11 files changed

+164
-23
lines changed

doc/api/training/sdp_versions/v1.0.0/smd_data_parallel_pytorch.rst

+1-1
Original file line numberDiff line numberDiff line change
@@ -155,7 +155,7 @@ PyTorch API
155155

156156
**Supported versions:**
157157

158-
- PyTorch 1.6
158+
- PyTorch 1.6.0
159159

160160

161161
.. function:: smdistributed.dataparallel.torch.distributed.is_available()

doc/api/training/sdp_versions/v1.0.0/smd_data_parallel_tensorflow.rst

+2-2
Original file line numberDiff line numberDiff line change
@@ -414,7 +414,7 @@ TensorFlow API
414414

415415
.. function:: smdistributed.dataparallel.tensorflow.DistributedOptimizer
416416

417-
Applicable if you use the ``tf.estimator`` API in TensorFlow 2.x (2.3).
417+
Applicable if you use the ``tf.estimator`` API in TensorFlow 2.x (2.3.1).
418418
419419
Construct a new ``DistributedOptimizer`` , which uses TensorFlow
420420
optimizer under the hood for computing single-process gradient values
@@ -489,7 +489,7 @@ TensorFlow API
489489

490490
.. function:: smdistributed.dataparallel.tensorflow.BroadcastGlobalVariablesHook
491491

492-
Applicable if you use the ``tf.estimator`` API in TensorFlow 2.x (2.3).
492+
Applicable if you use the ``tf.estimator`` API in TensorFlow 2.x (2.3.1).
493493

494494

495495
``SessionRunHook`` that will broadcast all global variables from root

doc/api/training/smd_model_parallel_release_notes/smd_model_parallel_change_log.md

+9-5
Original file line numberDiff line numberDiff line change
@@ -8,14 +8,18 @@
88

99
### PyTorch
1010

11-
#### Add support for PyTorch 1.7
11+
#### Add support for PyTorch 1.7.1
1212

13-
- Adds support for `gradient_as_bucket_view` (PyTorch 1.7 only), `find_unused_parameters` (PyTorch 1.7 only) and `broadcast_buffers` options to `smp.DistributedModel`. These options behave the same as the corresponding options (with the same names) in
13+
- Adds support for `gradient_as_bucket_view` (PyTorch 1.7.1 only), `find_unused_parameters` (PyTorch 1.7.1 only) and `broadcast_buffers` options to `smp.DistributedModel`. These options behave the same as the corresponding options (with the same names) in
1414
`torch.DistributedDataParallel` API. Please refer to the [SageMaker distributed model parallel API documentation](https://sagemaker.readthedocs.io/en/stable/api/training/smd_model_parallel_pytorch.html#smp.DistributedModel) for more information.
1515

16-
- Adds support for `join` (PyTorch 1.7 only) context manager, which is to be used in conjunction with an instance of `smp.DistributedModel` to be able to train with uneven inputs across participating processes.
16+
- Adds support for `join` (PyTorch 1.7.1 only) context manager, which is to be used in conjunction with an instance of `smp.DistributedModel` to be able to train with uneven inputs across participating processes.
1717

18-
- Adds support for `_register_comm_hook` (PyTorch 1.7 only) which will register the callable as a communication hook for DDP. NOTE: Like in DDP, this is an experimental API and subject to change.
18+
- Adds support for `_register_comm_hook` (PyTorch 1.7.1 only) which will register the callable as a communication hook for DDP. NOTE: Like in DDP, this is an experimental API and subject to change.
19+
20+
### Tensorflow
21+
22+
- Adds support for Tensorflow 2.4.1
1923

2024
## Bug Fixes
2125

@@ -32,7 +36,7 @@ regular dicts.
3236

3337
### PyTorch
3438

35-
- A performance regression was observed when training on SMP with PyTorch 1.7.1 compared to 1.6. The rootcause was found to be the slowdown in performance of `.grad` method calls in PyTorch 1.7.1 compared to 1.6. Please see the related discussion: https://github.com/pytorch/pytorch/issues/50636.
39+
- A performance regression was observed when training on SMP with PyTorch 1.7.1 compared to 1.6.0. The rootcause was found to be the slowdown in performance of `.grad` method calls in PyTorch 1.7.1 compared to 1.6.0. Please see the related discussion: https://github.com/pytorch/pytorch/issues/50636.
3640

3741

3842
# Sagemaker Distributed Model Parallel 1.1.0 Release Notes

doc/api/training/smp_versions/v1.1.0/smd_model_parallel_tensorflow.rst

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
TensorFlow API
22
==============
33

4-
**Supported version: 2.3**
4+
**Supported version: 2.3.1**
55

66
**Important**: This API document assumes you use the following import statement in your training scripts.
77

doc/api/training/smp_versions/v1.2.0/smd_model_parallel_pytorch.rst

+3-3
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
PyTorch API
77
===========
88

9-
**Supported versions: 1.7.1, 1.6**
9+
**Supported versions: 1.7.1, 1.6.0**
1010

1111
This API document assumes you use the following import statements in your training scripts.
1212

@@ -159,7 +159,7 @@ This API document assumes you use the following import statements in your traini
159159
This parameter is forwarded to the underlying ``DistributedDataParallel`` wrapper.
160160
Please see: `broadcast_buffer <https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel>`__.
161161

162-
- ``gradient_as_bucket_view (PyTorch 1.7 only)`` (default: False): To be
162+
- ``gradient_as_bucket_view (PyTorch 1.7.1 only)`` (default: False): To be
163163
used with ``ddp=True``. This parameter is forwarded to the underlying
164164
``DistributedDataParallel`` wrapper. Please see `gradient_as_bucket_view <https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel>`__.
165165

@@ -257,7 +257,7 @@ This API document assumes you use the following import statements in your traini
257257

258258
.. function:: join( )
259259

260-
**Available for PyTorch 1.7 only**
260+
**Available for PyTorch 1.7.1 only**
261261

262262
A context manager to be used in conjunction with an instance of
263263
``smp.DistributedModel`` to be able to train with uneven inputs across

doc/api/training/smp_versions/v1.2.0/smd_model_parallel_tensorflow.rst

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
TensorFlow API
22
==============
33

4-
**Supported version: 2.3**
4+
**Supported version: 2.4.1, 2.3.1**
55

66
**Important**: This API document assumes you use the following import statement in your training scripts.
77

src/sagemaker/estimator.py

+3
Original file line numberDiff line numberDiff line change
@@ -349,6 +349,9 @@ def __init__(
349349
self.profiler_config = profiler_config
350350
self.disable_profiler = disable_profiler
351351

352+
if not _region_supports_profiler(self.sagemaker_session.boto_region_name):
353+
self.disable_profiler = True
354+
352355
self.profiler_rule_configs = None
353356
self.profiler_rules = None
354357
self.debugger_rules = None

src/sagemaker/local/local_session.py

+18-5
Original file line numberDiff line numberDiff line change
@@ -426,17 +426,27 @@ def invoke_endpoint(
426426
CustomAttributes=None,
427427
TargetModel=None,
428428
TargetVariant=None,
429+
InferenceId=None,
429430
):
430431
"""Invoke the endpoint.
431432
432433
Args:
433-
Body:
434-
EndpointName:
435-
Accept: (Default value = None)
436-
CustomAttributes: (Default value = None)
434+
Body: Input data for which you want the model to provide inference.
435+
EndpointName: The name of the endpoint that you specified when you
436+
created the endpoint using the CreateEndpoint API.
437+
ContentType: The MIME type of the input data in the request body (Default value = None)
438+
Accept: The desired MIME type of the inference in the response (Default value = None)
439+
CustomAttributes: Provides additional information about a request for an inference
440+
submitted to a model hosted at an Amazon SageMaker endpoint (Default value = None)
441+
TargetModel: The model to request for inference when invoking a multi-model endpoint
442+
(Default value = None)
443+
TargetVariant: Specify the production variant to send the inference request to when
444+
invoking an endpoint that is running two or more variants (Default value = None)
445+
InferenceId: If you provide a value, it is added to the captured data when you enable
446+
data capture on the endpoint (Default value = None)
437447
438448
Returns:
439-
449+
object: Inference for the given input.
440450
"""
441451
url = "http://localhost:%s/invocations" % self.serving_port
442452
headers = {}
@@ -456,6 +466,9 @@ def invoke_endpoint(
456466
if TargetVariant is not None:
457467
headers["X-Amzn-SageMaker-Target-Variant"] = TargetVariant
458468

469+
if InferenceId is not None:
470+
headers["X-Amzn-SageMaker-Inference-Id"] = InferenceId
471+
459472
r = self.http.request("POST", url, body=Body, preload_content=False, headers=headers)
460473

461474
return {"Body": r, "ContentType": Accept}

src/sagemaker/predictor.py

+16-5
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,9 @@ def __init__(
9595
self._model_names = self._get_model_names()
9696
self._context = None
9797

98-
def predict(self, data, initial_args=None, target_model=None, target_variant=None):
98+
def predict(
99+
self, data, initial_args=None, target_model=None, target_variant=None, inference_id=None
100+
):
99101
"""Return the inference from the specified endpoint.
100102
101103
Args:
@@ -111,8 +113,10 @@ def predict(self, data, initial_args=None, target_model=None, target_variant=Non
111113
in case of a multi model endpoint. Does not apply to endpoints hosting
112114
single model (Default: None)
113115
target_variant (str): The name of the production variant to run an inference
114-
request on (Default: None). Note that the ProductionVariant identifies the model
115-
you want to host and the resources you want to deploy for hosting it.
116+
request on (Default: None). Note that the ProductionVariant identifies the
117+
model you want to host and the resources you want to deploy for hosting it.
118+
inference_id (str): If you provide a value, it is added to the captured data
119+
when you enable data capture on the endpoint (Default: None).
116120
117121
Returns:
118122
object: Inference for the given input. If a deserializer was specified when creating
@@ -121,7 +125,9 @@ def predict(self, data, initial_args=None, target_model=None, target_variant=Non
121125
as is.
122126
"""
123127

124-
request_args = self._create_request_args(data, initial_args, target_model, target_variant)
128+
request_args = self._create_request_args(
129+
data, initial_args, target_model, target_variant, inference_id
130+
)
125131
response = self.sagemaker_session.sagemaker_runtime_client.invoke_endpoint(**request_args)
126132
return self._handle_response(response)
127133

@@ -131,7 +137,9 @@ def _handle_response(self, response):
131137
content_type = response.get("ContentType", "application/octet-stream")
132138
return self.deserializer.deserialize(response_body, content_type)
133139

134-
def _create_request_args(self, data, initial_args=None, target_model=None, target_variant=None):
140+
def _create_request_args(
141+
self, data, initial_args=None, target_model=None, target_variant=None, inference_id=None
142+
):
135143
"""Placeholder docstring"""
136144
args = dict(initial_args) if initial_args else {}
137145

@@ -150,6 +158,9 @@ def _create_request_args(self, data, initial_args=None, target_model=None, targe
150158
if target_variant:
151159
args["TargetVariant"] = target_variant
152160

161+
if inference_id:
162+
args["InferenceId"] = inference_id
163+
153164
data = self.serializer.serialize(data)
154165

155166
args["Body"] = data
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
# Copyright 2019-2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License"). You
4+
# may not use this file except in compliance with the License. A copy of
5+
# the License is located at
6+
#
7+
# http://aws.amazon.com/apache2.0/
8+
#
9+
# or in the "license" file accompanying this file. This file is
10+
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
11+
# ANY KIND, either express or implied. See the License for the specific
12+
# language governing permissions and limitations under the License.
13+
from __future__ import absolute_import
14+
15+
import os
16+
import pytest
17+
18+
import tests.integ
19+
import tests.integ.timeout
20+
21+
from sagemaker import image_uris
22+
from sagemaker.model import Model
23+
from sagemaker.predictor import Predictor
24+
from sagemaker.serializers import CSVSerializer
25+
from sagemaker.utils import unique_name_from_base
26+
27+
from tests.integ import DATA_DIR
28+
29+
30+
ROLE = "SageMakerRole"
31+
INSTANCE_COUNT = 1
32+
INSTANCE_TYPE = "ml.c5.xlarge"
33+
TEST_CSV_DATA = "42,42,42,42,42,42,42"
34+
XGBOOST_DATA_PATH = os.path.join(DATA_DIR, "xgboost_model")
35+
36+
37+
@pytest.yield_fixture(scope="module")
38+
def endpoint_name(sagemaker_session):
39+
endpoint_name = unique_name_from_base("model-inference-id-integ")
40+
xgb_model_data = sagemaker_session.upload_data(
41+
path=os.path.join(XGBOOST_DATA_PATH, "xgb_model.tar.gz"),
42+
key_prefix="integ-test-data/xgboost/model",
43+
)
44+
45+
xgb_image = image_uris.retrieve(
46+
"xgboost",
47+
sagemaker_session.boto_region_name,
48+
version="1",
49+
image_scope="inference",
50+
)
51+
52+
with tests.integ.timeout.timeout_and_delete_endpoint_by_name(
53+
endpoint_name=endpoint_name, sagemaker_session=sagemaker_session, hours=2
54+
):
55+
xgb_model = Model(
56+
model_data=xgb_model_data,
57+
image_uri=xgb_image,
58+
name=endpoint_name, # model name
59+
role=ROLE,
60+
sagemaker_session=sagemaker_session,
61+
)
62+
xgb_model.deploy(INSTANCE_COUNT, INSTANCE_TYPE, endpoint_name=endpoint_name)
63+
yield endpoint_name
64+
65+
66+
def test_predict_with_inference_id(sagemaker_session, endpoint_name):
67+
predictor = Predictor(
68+
endpoint_name=endpoint_name,
69+
sagemaker_session=sagemaker_session,
70+
serializer=CSVSerializer(),
71+
)
72+
73+
# Validate that no exception is raised when the target_variant is specified.
74+
response = predictor.predict(TEST_CSV_DATA, inference_id="foo")
75+
assert response
76+
77+
78+
def test_invoke_endpoint_with_inference_id(sagemaker_session, endpoint_name):
79+
response = sagemaker_session.sagemaker_runtime_client.invoke_endpoint(
80+
EndpointName=endpoint_name,
81+
Body=TEST_CSV_DATA,
82+
ContentType="text/csv",
83+
Accept="text/csv",
84+
InferenceId="foo",
85+
)
86+
assert response

tests/unit/test_predictor.py

+24
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
RETURN_VALUE = 0
3232
CSV_RETURN_VALUE = "1,2,3\r\n"
3333
PRODUCTION_VARIANT_1 = "PRODUCTION_VARIANT_1"
34+
INFERENCE_ID = "inference-id"
3435

3536
ENDPOINT_DESC = {"EndpointArn": "foo", "EndpointConfigName": ENDPOINT}
3637

@@ -98,6 +99,29 @@ def test_predict_call_with_target_variant():
9899
assert result == RETURN_VALUE
99100

100101

102+
def test_predict_call_with_inference_id():
103+
sagemaker_session = empty_sagemaker_session()
104+
predictor = Predictor(ENDPOINT, sagemaker_session)
105+
106+
data = "untouched"
107+
result = predictor.predict(data, inference_id=INFERENCE_ID)
108+
109+
assert sagemaker_session.sagemaker_runtime_client.invoke_endpoint.called
110+
111+
expected_request_args = {
112+
"Accept": DEFAULT_ACCEPT,
113+
"Body": data,
114+
"ContentType": DEFAULT_CONTENT_TYPE,
115+
"EndpointName": ENDPOINT,
116+
"InferenceId": INFERENCE_ID,
117+
}
118+
119+
call_args, kwargs = sagemaker_session.sagemaker_runtime_client.invoke_endpoint.call_args
120+
assert kwargs == expected_request_args
121+
122+
assert result == RETURN_VALUE
123+
124+
101125
def test_multi_model_predict_call():
102126
sagemaker_session = empty_sagemaker_session()
103127
predictor = Predictor(ENDPOINT, sagemaker_session)

0 commit comments

Comments
 (0)