Skip to content

fix: support estimator output path parameterization #3111

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 127 commits into from
May 25, 2022
Merged

fix: support estimator output path parameterization #3111

merged 127 commits into from
May 25, 2022

Conversation

jerrypeng7773
Copy link
Contributor

@jerrypeng7773 jerrypeng7773 commented May 13, 2022 •
edited
Loading

Issue #, if available:

#3104
#3078

Description of changes:

support estimator output_path parameterization

Testing done:

unit + integ

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

  • I have read the CONTRIBUTING doc
  • I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
  • I used the commit message format described in CONTRIBUTING
  • I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
  • I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
  • I have checked that my tests are not configured for a specific region or account (if appropriate)
  • I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

navinsoni and others added 30 commits March 1, 2022 14:18
@navinsoni @shreyapandit
change: update code to get commit_id in codepipeline (aws#2961)
f166b60
@jeniyat @shreyapandit
feature: Data Serializer (aws#2956)
086258d
@qidewenwhen @bencrabtree
@navinsoni @jeniyat
change: reorganize test files for workflow (aws#2960)
a39b750 
Co-authored-by: Ben Crabtree <[email protected]>
Co-authored-by: Navin Soni <[email protected]>
Co-authored-by: Jeniya Tabassum <[email protected]>
Co-authored-by: Dewen Qi <[email protected]>
@Qingzi-Lan @bencrabtree
@navinsoni @jeniyat
feature: TensorFlow 2.4 for Neo (aws#2861)
28fd737 
Co-authored-by: Ben Crabtree <[email protected]>
Co-authored-by: Navin Soni <[email protected]>
Co-authored-by: Jeniya Tabassum <[email protected]>
@staubhp
fix: Remove sagemaker_job_name from hyperparameters in TrainingStep (a…
20df3d7 
…ws#2950)

Co-authored-by: Payton Staub <[email protected]>
@jeniyat
fix: Style update in DataSerializer (aws#2962)
b9f90dc
@mchoi8739
documentation: smddp doc update (aws#2968)
6db3774
@shreyapandit
fix: container env generation for S3 URI and add test for the same (a…
d610bfb 
…ws#2971)
@mchoi8739
documentation: update sagemaker training compiler docstring (aws#2969)
169dffd
@ahsan-z-khan @bencrabtree
@navinsoni @jeniyat @qidewenwhen @Qingzi-Lan @staubhp
feat: Python 3.9 for readthedocs (aws#2973)
4325fcd 
Co-authored-by: Ben Crabtree <[email protected]>
Co-authored-by: Navin Soni <[email protected]>
Co-authored-by: Jeniya Tabassum <[email protected]>
Co-authored-by: Dewen Qi <[email protected]>
Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: qidewenwhen <[email protected]>
Co-authored-by: Qingzi-Lan <[email protected]>
Co-authored-by: Payton Staub <[email protected]>
@jerrypeng7773
prepare release v2.78.0
3baf513
@jerrypeng7773
update development version to v2.78.1.dev0
c0c2b9c
@mchoi8739 @ahsan-z-khan
@shreyapandit @mufaddal-rohawala @BasilBeirouti @staubhp @mohamed-ali @jeniyat @sreedes @navinns @Zhankuil @navinsoni @xgchena @JGuinegagne @navaj0 @yzhu0 @Qingzi-Lan @bencrabtree @qidewenwhen @xchen909 @tuliocasagrande @HappyAmazonian @jerrypeng7773
documentation: the SageMaker distributed data parallel v1.4.0 release (…
b74736f 
…aws#2980)

Co-authored-by: Ahsan Khan <[email protected]>
Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: Shreya Pandit <[email protected]>
Co-authored-by: Basil Beirouti <[email protected]>
Co-authored-by: Mufaddal Rohawala <[email protected]>
Co-authored-by: Basil Beirouti <[email protected]>
Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: Mohamed Ali Jamaoui <[email protected]>
Co-authored-by: ci <ci>
Co-authored-by: Jeniya Tabassum <[email protected]>
Co-authored-by: sreedes <[email protected]>
Co-authored-by: Navin Soni <[email protected]>
Co-authored-by: Miyoung <[email protected]>
Co-authored-by: Ameen Khan <[email protected]>
Co-authored-by: Zhankui Lu <[email protected]>
Co-authored-by: Navin Soni <[email protected]>
Co-authored-by: Xiaoguang Chen <[email protected]>
Co-authored-by: Jonathan Guinegagne <[email protected]>
Co-authored-by: Zhankui Lu <[email protected]>
Co-authored-by: Yifei Zhu <[email protected]>
Co-authored-by: Qingzi-Lan <[email protected]>
Co-authored-by: Ben Crabtree <[email protected]>
Co-authored-by: Dewen Qi <[email protected]>
Co-authored-by: qidewenwhen <[email protected]>
Co-authored-by: Xinghan Chen <[email protected]>
Co-authored-by: Tulio Casagrande <[email protected]>
Co-authored-by: HappyAmazonian <[email protected]>
@evakravi @bencrabtree
@navinsoni @jeniyat @qidewenwhen @Qingzi-Lan @staubhp @mchoi8739 @shreyapandit @ahsan-z-khan @jerrypeng7773
feat: custom base job name for jumpstart models/estimators (aws#2970)
be831b4 
Co-authored-by: Ben Crabtree <[email protected]>
Co-authored-by: Navin Soni <[email protected]>
Co-authored-by: Jeniya Tabassum <[email protected]>
Co-authored-by: Dewen Qi <[email protected]>
Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: qidewenwhen <[email protected]>
Co-authored-by: Qingzi-Lan <[email protected]>
Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: Miyoung <[email protected]>
Co-authored-by: Shreya Pandit <[email protected]>
Co-authored-by: Ahsan Khan <[email protected]>
@jeniyat @jerrypeng7773
feature: Inferentia Neuron support for HuggingFace (aws#2976)
588dd69
@jerrypeng7773
prepare release v2.79.0
80f3054
@jerrypeng7773
update development version to v2.79.1.dev0
a57ad9d
@evakravi @jerrypeng7773
fix: jumpstart docs network isolation (aws#2989)
f6689a3
@gradientsky @jerrypeng7773
feature: AutoGluon 0.3.2 and 0.4.0 image_uris (aws#2991)
4440b5b
@samadwar @bencrabtree
@navinsoni @jeniyat @qidewenwhen @Qingzi-Lan @staubhp @mchoi8739 @shreyapandit @ahsan-z-khan @jerrypeng7773
feature: Support for remote docker host (aws#2864)
eab7723 
Co-authored-by: Ben Crabtree <[email protected]>
Co-authored-by: Navin Soni <[email protected]>
Co-authored-by: Jeniya Tabassum <[email protected]>
Co-authored-by: Dewen Qi <[email protected]>
Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: qidewenwhen <[email protected]>
Co-authored-by: Qingzi-Lan <[email protected]>
Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: EC2 Default User <[email protected]>
Co-authored-by: Miyoung <[email protected]>
Co-authored-by: Shreya Pandit <[email protected]>
Co-authored-by: Ahsan Khan <[email protected]>
@mufaddal-rohawala @ahsan-z-khan @jerrypeng7773
fix: gpu integs CapacityError - fallback to available compute (aws#3004)
c87a887 
Co-authored-by: Ahsan Khan <[email protected]>
@saimidu @mufaddal-rohawala @jerrypeng7773
feature: Add support for TF 2.6.3 (aws#3006)
f49061b 
Co-authored-by: Mufaddal Rohawala <[email protected]>
@Qingzi-Lan @mufaddal-rohawala @jerrypeng7773
feature: TF242 ioc support (aws#2982)
a629f2f 
Co-authored-by: Mufaddal Rohawala <[email protected]>
@saimidu @mufaddal-rohawala @jerrypeng7773
feature: Add support for TF 2.8 (aws#3000)
f40ace0 
* feature: Add support for TF 2.7 and TF 2.8

* Correct TF 2.7 patch version used in tests

* Correct TF 2.7 patch version for SMDDP TF versions

* Only add entries for TF 2.8

* Fix test options for TF 2.8 for fw_utils

* Update tensorflow.json

Co-authored-by: Mufaddal Rohawala <[email protected]>
@mchoi8739 @navinsoni
@jeniyat @qidewenwhen @bencrabtree @Qingzi-Lan @staubhp @shreyapandit @ahsan-z-khan @mufaddal-rohawala @jerrypeng7773
documentation: sagemaker distributed model parallel 1.7.0 doc (aws#2992)
c9539bd 
* change: update code to get commit_id in codepipeline (aws#2961)

* feature: Data Serializer (aws#2956)

* change: reorganize test files for workflow (aws#2960)

Co-authored-by: Ben Crabtree <[email protected]>
Co-authored-by: Navin Soni <[email protected]>
Co-authored-by: Jeniya Tabassum <[email protected]>
Co-authored-by: Dewen Qi <[email protected]>

* feature: TensorFlow 2.4 for Neo (aws#2861)

Co-authored-by: Ben Crabtree <[email protected]>
Co-authored-by: Navin Soni <[email protected]>
Co-authored-by: Jeniya Tabassum <[email protected]>

* fix: Remove sagemaker_job_name from hyperparameters in TrainingStep (aws#2950)

Co-authored-by: Payton Staub <[email protected]>

* fix: Style update in DataSerializer (aws#2962)

* documentation: smddp doc update (aws#2968)

* fix: container env generation for S3 URI and add test for the same (aws#2971)

* documentation: update sagemaker training compiler docstring (aws#2969)

* feat: Python 3.9 for readthedocs (aws#2973)

Co-authored-by: Ben Crabtree <[email protected]>
Co-authored-by: Navin Soni <[email protected]>
Co-authored-by: Jeniya Tabassum <[email protected]>
Co-authored-by: Dewen Qi <[email protected]>
Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: qidewenwhen <[email protected]>
Co-authored-by: Qingzi-Lan <[email protected]>
Co-authored-by: Payton Staub <[email protected]>

* fix doc structure

* archive 1.6.0 doc

* add new args, refs, and links

* fix version number

* incorp eng feedback, update docstrings, improve xref

* Trigger Build

* minor fix, trigger build again

* fix typo

Co-authored-by: Navin Soni <[email protected]>
Co-authored-by: Jeniya Tabassum <[email protected]>
Co-authored-by: qidewenwhen <[email protected]>
Co-authored-by: Ben Crabtree <[email protected]>
Co-authored-by: Dewen Qi <[email protected]>
Co-authored-by: Qingzi-Lan <[email protected]>
Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: Shreya Pandit <[email protected]>
Co-authored-by: Ahsan Khan <[email protected]>
Co-authored-by: Mufaddal Rohawala <[email protected]>
@mufaddal-rohawala @jerrypeng7773
fix: gpu integs CapacityError - fallback to available compute (aws#3008)
df29ea3
@arjkesh @bencrabtree
@navinsoni @jeniyat @qidewenwhen @Qingzi-Lan @staubhp @mchoi8739 @shreyapandit @ahsan-z-khan @mufaddal-rohawala @jerrypeng7773
feature: Add support for TF2.7 (aws#2958)
4079ea1 
Co-authored-by: Ben Crabtree <[email protected]>
Co-authored-by: Navin Soni <[email protected]>
Co-authored-by: Jeniya Tabassum <[email protected]>
Co-authored-by: Dewen Qi <[email protected]>
Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: qidewenwhen <[email protected]>
Co-authored-by: Qingzi-Lan <[email protected]>
Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: Miyoung <[email protected]>
Co-authored-by: Shreya Pandit <[email protected]>
Co-authored-by: Ahsan Khan <[email protected]>
Co-authored-by: Mufaddal Rohawala <[email protected]>
@IvyBazan @ahsan-z-khan
@shreyapandit @mufaddal-rohawala @jerrypeng7773
change: Add JumpStart model table build notification (aws#2997)
f2a0379 
Co-authored-by: Ahsan Khan <[email protected]>
Co-authored-by: Shreya Pandit <[email protected]>
Co-authored-by: Mufaddal Rohawala <[email protected]>
@mohamed-ali @jerrypeng7773
fix: Align max_wait definitions in EstimaorBase and Estimator (aws#3009)
e22e04f 
Co-authored-by: Mohamed Ali Jamaoui <[email protected]>
@jerrypeng7773
prepare release v2.80.0
9474bd3
@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 80d5b4f
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: 80d5b4f
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 80d5b4f
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: 80d5b4f
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: 80d5b4f
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 1100651
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 1100651
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: 1100651
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: 1100651
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 1100651
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: 1100651
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: 1100651
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: ceadb04
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: ceadb04
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: ceadb04
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: ceadb04
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: ceadb04
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: ceadb04
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: ceadb04
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: ceadb04
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: ceadb04
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: ceadb04
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: ceadb04
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@navinsoni navinsoni merged commit d61dec6 into aws:master May 25, 2022
@lkev lkev mentioned this pull request Jun 1, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@qidewenwhen qidewenwhen qidewenwhen approved these changes

@shreyapandit shreyapandit shreyapandit approved these changes

@navinsoni navinsoni navinsoni approved these changes

@nmadan nmadan nmadan approved these changes

@navaj0 navaj0 Awaiting requested review from navaj0

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.