Skip to content

Commit ed9131b

Browse files
author
Eugene Teoh
committed
Merge remote-tracking branch 'upstream/dev' into repack_output
2 parents 0489b59 + 4325fcd commit ed9131b

File tree

207 files changed

+20064
-4124
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

207 files changed

+20064
-4124
lines changed

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,4 +28,5 @@ venv/
2828
.docker/
2929
env/
3030
.vscode/
31+
**/tmp
3132
.python-version

‎.readthedocs.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
version: 2
66

77
python:
8-
version: 3.6
8+
version: 3.9
99
install:
1010
- method: pip
1111
path: .

‎CHANGELOG.md

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,124 @@
11
# Changelog
22

3+
## v2.77.1 (2022-02-25)
4+
5+
### Features
6+
7+
* default repack encryption
8+
* support large pipeline
9+
* add support for pytorch 1.10.0
10+
11+
### Documentation Changes
12+
13+
* SageMaker model parallel library 1.6.0 API doc
14+
15+
### Bug Fixes and Other Changes
16+
17+
* jumpstart model table
18+
19+
## v2.77.0 (2022-02-22)
20+
21+
### Features
22+
23+
* override jumpstart content bucket
24+
* jumpstart model id suggestions
25+
* adding customer metadata support to registermodel step
26+
27+
### Bug Fixes and Other Changes
28+
29+
* Improve Pipeline workflow unit test branch coverage
30+
* update lineage_trial_compoment get pipeline execution arn
31+
* Add lineage doc
32+
* Support primitive types for left value of ConditionSteps
33+
34+
## v2.76.0 (2022-02-17)
35+
36+
### Features
37+
38+
* Add FailStep Support for Sagemaker Pipeline
39+
40+
### Bug Fixes and Other Changes
41+
42+
* use recommended inference image uri from Neo API
43+
* pin test dependencies
44+
* Add exception in test_action
45+
* Update Static Endpoint
46+
* Add CMH to the non-P3 list
47+
48+
### Documentation Changes
49+
50+
* Support for generation of Jumpstart model table on build
51+
52+
## v2.75.1 (2022-02-08)
53+
54+
### Bug Fixes and Other Changes
55+
56+
* Add CMH to the non-P3 list
57+
58+
## v2.75.0 (2022-02-05)
59+
60+
### Features
61+
62+
* JumpStart Integration
63+
* Adds support for async inference
64+
* Update instance types for integ test
65+
66+
### Bug Fixes and Other Changes
67+
68+
* Revert "feature: CompilationStep support for Sagemaker Pipelines
69+
* gpu use p3/p2 per avail for region
70+
* jumpstart typo
71+
* pin pytest-xdist to avoid release failures
72+
* set sagemaker_connection and image_uri in register method
73+
* update to incorporate black v22, pin tox versions
74+
* Add deprecation warning in Clarify DataConfig
75+
76+
### Documentation Changes
77+
78+
* Jumpstart doc strings and added new sections
79+
* Add Jumpstart support documentation
80+
81+
## v2.74.0 (2022-01-26)
82+
83+
### Features
84+
85+
* Add support for SageMaker lineage queries context
86+
87+
### Bug Fixes and Other Changes
88+
89+
* support specifying a facet by its column index
90+
91+
### Documentation Changes
92+
93+
* more documentation for serverless inference
94+
95+
## v2.73.0 (2022-01-19)
96+
97+
### Features
98+
99+
* Add EMRStep support in Sagemaker pipeline
100+
* Adds Lineage queries in artifact, context and trial components
101+
* Add support for SageMaker lineage queries in action
102+
* Adds support for Serverless inference
103+
* support checkpoint to be passed from estimator
104+
* support JsonGet/Join parameterization in tuning step Hyperparameters
105+
* Support model pipelines in CreateModelStep
106+
* enable python 3.9
107+
* Add models_v2 under lineage context
108+
109+
### Bug Fixes and Other Changes
110+
111+
* allow kms_key to be passed for processing step
112+
* Remove duplicate vertex/edge in query lineage
113+
* update pricing link
114+
* Update CHANGELOG.md
115+
* fixes unnecessary session call while generating pipeline definition for lambda step
116+
117+
### Documentation Changes
118+
119+
* Enhance smddp 1.2.2 doc
120+
* Document the available ExecutionVariables
121+
3122
## v2.72.3 (2022-01-10)
4123

5124
### Features

‎README.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,7 @@ SageMaker Python SDK is tested on:
9090
- Python 3.6
9191
- Python 3.7
9292
- Python 3.8
93+
- Python 3.9
9394

9495
AWS Permissions
9596
~~~~~~~~~~~~~~~

‎VERSION

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
2.72.4.dev0
1+
2.77.2.dev0

‎doc/api/inference/async_inference.rst

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
Async Inference
2+
-----------------
3+
4+
This module contains classes related to Amazon Sagemaker Async Inference
5+
6+
.. automodule:: sagemaker.async_inference.async_inference_config
7+
:members:
8+
:undoc-members:
9+
:show-inheritance:
10+
11+
.. automodule:: sagemaker.async_inference.async_inference_response
12+
:members:
13+
:undoc-members:
14+
:show-inheritance:
15+
16+
.. automodule:: sagemaker.async_inference.waiter_config
17+
:members:
18+
:undoc-members:
19+
:show-inheritance:

‎doc/api/inference/predictor_async.rst

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
AsyncPredictor
2+
--------------------
3+
4+
Make async predictions against SageMaker endpoints with Python objects
5+
6+
.. autoclass:: sagemaker.predictor_async.AsyncPredictor
7+
:members:
8+
:undoc-members:
9+
:show-inheritance:

‎doc/api/inference/serverless.rst

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
Serverless Inference
2+
---------------------
3+
4+
This module contains classes related to Amazon Sagemaker Serverless Inference
5+
6+
.. automodule:: sagemaker.serverless.serverless_inference_config
7+
:members:
8+
:undoc-members:
9+
:show-inheritance:

‎doc/api/training/sdp_versions/latest/smd_data_parallel_tensorflow.rst

Lines changed: 37 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -243,16 +243,25 @@ TensorFlow API
243243

244244
.. function:: smdistributed.dataparallel.tensorflow.allreduce(tensor, param_index, num_params, compression=Compression.none, op=ReduceOp.AVERAGE)
245245

246-
Performs an all-reduce operation on a tensor (``tf.Tensor``).
246+
Performs an ``allreduce`` operation on a tensor (``tf.Tensor``).
247+
248+
The ``smdistributed.dataparallel`` package's AllReduce API for TensorFlow to allreduce
249+
gradient tensors. By default, ``smdistributed.dataparallel`` allreduce averages the
250+
gradient tensors across participating workers.
251+
252+
.. note::
253+
254+
:class:`smdistributed.dataparallel.tensorflow.allreduce()` should
255+
only be used to allreduce gradient tensors.
256+
For other (non-gradient) tensors, you must use
257+
:class:`smdistributed.dataparallel.tensorflow.oob_allreduce()`.
258+
If you use :class:`smdistributed.dataparallel.tensorflow.allreduce()`
259+
for non-gradient tensors,
260+
the distributed training job might stall or stop.
247261

248-
``smdistributed.dataparallel`` AllReduce API can be used for all
249-
reducing gradient tensors or any other tensors. By
250-
default, ``smdistributed.dataparallel`` AllReduce averages the
251-
tensors across the participating workers.
252-
​
253262
**Inputs:**
254263

255-
- ``tensor (tf.Tensor)(required)``: The tensor to be all-reduced. The shape of the input must be identical across all ranks.
264+
- ``tensor (tf.Tensor)(required)``: The tensor to be allreduced. The shape of the input must be identical across all ranks.
256265
- ``param_index (int)(required):`` 0 if you are reducing a single tensor. Index of the tensor if you are reducing a list of tensors.
257266
- ``num_params (int)(required):`` len(tensor).
258267
- ``compression (smdistributed.dataparallel.tensorflow.Compression)(optional)``: Compression algorithm used to reduce the amount of data sent and received by each worker node. Defaults to not using compression.
@@ -306,9 +315,9 @@ TensorFlow API
306315

307316
.. function:: smdistributed.dataparallel.tensorflow.oob_allreduce(tensor, compression=Compression.none, op=ReduceOp.AVERAGE)
308317

309-
OutOfBand (oob) AllReduce is simplified AllReduce function for use cases
318+
Out-of-band (oob) AllReduce is simplified AllReduce function for use-cases
310319
such as calculating total loss across all the GPUs in the training.
311-
oob_allreduce average the tensors, as reduction operation, across the
320+
``oob_allreduce`` average the tensors, as reduction operation, across the
312321
worker nodes.
313322

314323
**Inputs:**
@@ -326,15 +335,25 @@ TensorFlow API
326335

327336
- ``None``
328337

329-
.. rubric:: Notes
330-
331-
``smdistributed.dataparallel.tensorflow.oob_allreduce``, in most
332-
cases, is ~2x slower
333-
than ``smdistributed.dataparallel.tensorflow.allreduce``  so it is not
334-
recommended to be used for performing gradient reduction during the
335-
training
336-
process. ``smdistributed.dataparallel.tensorflow.oob_allreduce`` internally
337-
uses NCCL AllReduce with ``ncclSum`` as the reduction operation.
338+
.. note::
339+
340+
In most cases, the :class:`smdistributed.dataparallel.tensorflow.oob_allreduce()`
341+
function is ~2x slower
342+
than :class:`smdistributed.dataparallel.tensorflow.allreduce()`. It is not
343+
recommended to use the :class:`smdistributed.dataparallel.tensorflow.oob_allreduce()`
344+
function for performing gradient
345+
reduction during the training process.
346+
``smdistributed.dataparallel.tensorflow.oob_allreduce`` internally
347+
uses NCCL AllReduce with ``ncclSum`` as the reduction operation.
348+
349+
.. note::
350+
351+
:class:`smdistributed.dataparallel.tensorflow.oob_allreduce()` should
352+
only be used to allreduce non-gradient tensors.
353+
If you use :class:`smdistributed.dataparallel.tensorflow.allreduce()`
354+
for non-gradient tensors,
355+
the distributed training job might stall or stop.
356+
To allreduce gradients, use :class:`smdistributed.dataparallel.tensorflow.allreduce()`.
338357

339358

340359
.. function:: smdistributed.dataparallel.tensorflow.overlap(tensor)

‎doc/api/utility/environment_variables.rst

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Environment Variables
2+
---------------------
3+
4+
.. automodule:: sagemaker.environment_variables
5+
:members:
6+
:undoc-members:
7+
:show-inheritance:

‎doc/api/utility/hyperparameters.rst

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Hyperparameters
2+
---------------
3+
4+
.. automodule:: sagemaker.hyperparameters
5+
:members:
6+
:undoc-members:
7+
:show-inheritance:

‎doc/api/utility/model_uris.rst

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Model URIs
2+
----------
3+
4+
.. automodule:: sagemaker.model_uris
5+
:members:
6+
:undoc-members:
7+
:show-inheritance:

‎doc/api/utility/script_uris.rst

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Script URIs
2+
-----------
3+
4+
.. automodule:: sagemaker.script_uris
5+
:members:
6+
:undoc-members:
7+
:show-inheritance:

‎doc/conf.py

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,18 @@
1010
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
1111
# ANY KIND, either express or implied. See the License for the specific
1212
# language governing permissions and limitations under the License.
13-
"""Placeholder docstring"""
13+
"""Configuration for generating readthedocs docstrings."""
1414
from __future__ import absolute_import
1515

1616
import pkg_resources
1717
from datetime import datetime
18+
import sys
19+
import os
1820

19-
project = u"sagemaker"
21+
sys.path.append(os.path.join(os.path.dirname(__file__), "."))
22+
from doc_utils.jumpstart_doc_utils import create_jumpstart_model_table # noqa: E402
23+
24+
project = "sagemaker"
2025
version = pkg_resources.require(project)[0].version
2126

2227
# Add any Sphinx extension module names here, as strings. They can be extensions
@@ -38,7 +43,7 @@
3843
source_suffix = ".rst" # The suffix of source filenames.
3944
master_doc = "index" # The master toctree document.
4045

41-
copyright = u"%s, Amazon" % datetime.now().year
46+
copyright = "%s, Amazon" % datetime.now().year
4247

4348
# The full version, including alpha/beta/rc tags.
4449
release = version
@@ -71,6 +76,12 @@
7176
# For Adobe Analytics
7277
html_js_files = [
7378
"https://a0.awsstatic.com/s_code/js/3.0/awshome_s_code.js",
79+
"https://cdn.datatables.net/1.10.23/js/jquery.dataTables.min.js",
80+
"js/datatable.js",
81+
]
82+
83+
html_css_files = [
84+
"https://cdn.datatables.net/1.10.23/css/jquery.dataTables.min.css",
7485
]
7586

7687
html_context = {"css_files": ["_static/theme_overrides.css"]}
@@ -83,3 +94,7 @@
8394

8495
# autosectionlabel
8596
autosectionlabel_prefix_document = True
97+
98+
99+
def setup(app):
100+
create_jumpstart_model_table()

‎doc/doc_utils/__init__.py

Whitespace-only changes.

‎doc/doc_utils/jumpstart.rst

Whitespace-only changes.

0 commit comments

Comments
 (0)