fix: Remove sagemaker_job_name from hyperparameters in TrainingStep #2950

staubhp · 2022-02-22T23:54:29Z

Issue #, if available:
2940

Description of changes:
Some framework classes add a sagemaker_job_name hyperparameter with a dynamic training job name. This is both inaccurate (the job name will be set when the step runs) and it breaks caching for those steps.

Testing done:
Unit, manual

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

codecov-commenter · 2022-02-23T00:12:38Z

Codecov Report

Merging #2950 (2dfde22) into dev (086258d) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##              dev    #2950   +/-   ##
=======================================
  Coverage   89.80%   89.80%           
=======================================
  Files         196      196           
  Lines       16563    16565    +2     
=======================================
+ Hits        14875    14877    +2     
  Misses       1688     1688

Impacted Files	Coverage Δ
src/sagemaker/workflow/steps.py	`97.81% <100.00%> (+0.01%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 086258d...2dfde22. Read the comment docs.

sagemaker-bot · 2022-02-23T00:21:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 298f965
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-23T01:13:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 298f965
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-23T02:25:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 298f965
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-02T18:43:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6ff0c7a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-02T19:13:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: dbddcc2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-02T19:34:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 6ff0c7a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-02T19:35:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 6ff0c7a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-02T19:50:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: dbddcc2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jerrypeng7773 · 2022-03-02T20:57:50Z

src/sagemaker/workflow/steps.py

@@ -301,6 +301,7 @@ def arguments(self) -> RequestType:
        )
        request_dict = self.estimator.sagemaker_session._get_train_request(**train_args)
        request_dict.pop("TrainingJobName")
+        request_dict["HyperParameters"].pop("sagemaker_job_name", None)


wonder if it is possible that this request dict does not have HyperParameters in it?

It will always be there, but let me add a safety check

sagemaker-bot · 2022-03-02T21:00:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: dbddcc2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-02T21:20:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: dbddcc2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-02T23:49:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: dbddcc2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-03T00:30:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 2dfde22
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-03T01:21:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 2dfde22
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-03T01:48:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 2dfde22
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

tests/unit/sagemaker/workflow/test_step_collections.py

shreyapandit

As in comments

…2950) Co-authored-by: Payton Staub <[email protected]>

* change: update code to get commit_id in codepipeline (#2961) * feature: Data Serializer (#2956) * change: reorganize test files for workflow (#2960) Co-authored-by: Ben Crabtree <[email protected]> Co-authored-by: Navin Soni <[email protected]> Co-authored-by: Jeniya Tabassum <[email protected]> Co-authored-by: Dewen Qi <[email protected]> * feature: TensorFlow 2.4 for Neo (#2861) Co-authored-by: Ben Crabtree <[email protected]> Co-authored-by: Navin Soni <[email protected]> Co-authored-by: Jeniya Tabassum <[email protected]> * fix: Remove sagemaker_job_name from hyperparameters in TrainingStep (#2950) Co-authored-by: Payton Staub <[email protected]> * fix: Style update in DataSerializer (#2962) * documentation: smddp doc update (#2968) * fix: container env generation for S3 URI and add test for the same (#2971) * documentation: update sagemaker training compiler docstring (#2969) * feat: Python 3.9 for readthedocs (#2973) Co-authored-by: Ben Crabtree <[email protected]> Co-authored-by: Navin Soni <[email protected]> Co-authored-by: Jeniya Tabassum <[email protected]> Co-authored-by: Dewen Qi <[email protected]> Co-authored-by: Payton Staub <[email protected]> Co-authored-by: qidewenwhen <[email protected]> Co-authored-by: Qingzi-Lan <[email protected]> Co-authored-by: Payton Staub <[email protected]> * fix doc structure * archive 1.6.0 doc * add new args, refs, and links * fix version number * incorp eng feedback, update docstrings, improve xref * Trigger Build * minor fix, trigger build again * fix typo Co-authored-by: Navin Soni <[email protected]> Co-authored-by: Jeniya Tabassum <[email protected]> Co-authored-by: qidewenwhen <[email protected]> Co-authored-by: Ben Crabtree <[email protected]> Co-authored-by: Dewen Qi <[email protected]> Co-authored-by: Qingzi-Lan <[email protected]> Co-authored-by: Payton Staub <[email protected]> Co-authored-by: Payton Staub <[email protected]> Co-authored-by: Shreya Pandit <[email protected]> Co-authored-by: Ahsan Khan <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]>

* change: update code to get commit_id in codepipeline (aws#2961) * feature: Data Serializer (aws#2956) * change: reorganize test files for workflow (aws#2960) Co-authored-by: Ben Crabtree <[email protected]> Co-authored-by: Navin Soni <[email protected]> Co-authored-by: Jeniya Tabassum <[email protected]> Co-authored-by: Dewen Qi <[email protected]> * feature: TensorFlow 2.4 for Neo (aws#2861) Co-authored-by: Ben Crabtree <[email protected]> Co-authored-by: Navin Soni <[email protected]> Co-authored-by: Jeniya Tabassum <[email protected]> * fix: Remove sagemaker_job_name from hyperparameters in TrainingStep (aws#2950) Co-authored-by: Payton Staub <[email protected]> * fix: Style update in DataSerializer (aws#2962) * documentation: smddp doc update (aws#2968) * fix: container env generation for S3 URI and add test for the same (aws#2971) * documentation: update sagemaker training compiler docstring (aws#2969) * feat: Python 3.9 for readthedocs (aws#2973) Co-authored-by: Ben Crabtree <[email protected]> Co-authored-by: Navin Soni <[email protected]> Co-authored-by: Jeniya Tabassum <[email protected]> Co-authored-by: Dewen Qi <[email protected]> Co-authored-by: Payton Staub <[email protected]> Co-authored-by: qidewenwhen <[email protected]> Co-authored-by: Qingzi-Lan <[email protected]> Co-authored-by: Payton Staub <[email protected]> * fix doc structure * archive 1.6.0 doc * add new args, refs, and links * fix version number * incorp eng feedback, update docstrings, improve xref * Trigger Build * minor fix, trigger build again * fix typo Co-authored-by: Navin Soni <[email protected]> Co-authored-by: Jeniya Tabassum <[email protected]> Co-authored-by: qidewenwhen <[email protected]> Co-authored-by: Ben Crabtree <[email protected]> Co-authored-by: Dewen Qi <[email protected]> Co-authored-by: Qingzi-Lan <[email protected]> Co-authored-by: Payton Staub <[email protected]> Co-authored-by: Payton Staub <[email protected]> Co-authored-by: Shreya Pandit <[email protected]> Co-authored-by: Ahsan Khan <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]>

…ws#2950) Co-authored-by: Payton Staub <[email protected]>

* change: update code to get commit_id in codepipeline (aws#2961) * feature: Data Serializer (aws#2956) * change: reorganize test files for workflow (aws#2960) Co-authored-by: Ben Crabtree <[email protected]> Co-authored-by: Navin Soni <[email protected]> Co-authored-by: Jeniya Tabassum <[email protected]> Co-authored-by: Dewen Qi <[email protected]> * feature: TensorFlow 2.4 for Neo (aws#2861) Co-authored-by: Ben Crabtree <[email protected]> Co-authored-by: Navin Soni <[email protected]> Co-authored-by: Jeniya Tabassum <[email protected]> * fix: Remove sagemaker_job_name from hyperparameters in TrainingStep (aws#2950) Co-authored-by: Payton Staub <[email protected]> * fix: Style update in DataSerializer (aws#2962) * documentation: smddp doc update (aws#2968) * fix: container env generation for S3 URI and add test for the same (aws#2971) * documentation: update sagemaker training compiler docstring (aws#2969) * feat: Python 3.9 for readthedocs (aws#2973) Co-authored-by: Ben Crabtree <[email protected]> Co-authored-by: Navin Soni <[email protected]> Co-authored-by: Jeniya Tabassum <[email protected]> Co-authored-by: Dewen Qi <[email protected]> Co-authored-by: Payton Staub <[email protected]> Co-authored-by: qidewenwhen <[email protected]> Co-authored-by: Qingzi-Lan <[email protected]> Co-authored-by: Payton Staub <[email protected]> * fix doc structure * archive 1.6.0 doc * add new args, refs, and links * fix version number * incorp eng feedback, update docstrings, improve xref * Trigger Build * minor fix, trigger build again * fix typo Co-authored-by: Navin Soni <[email protected]> Co-authored-by: Jeniya Tabassum <[email protected]> Co-authored-by: qidewenwhen <[email protected]> Co-authored-by: Ben Crabtree <[email protected]> Co-authored-by: Dewen Qi <[email protected]> Co-authored-by: Qingzi-Lan <[email protected]> Co-authored-by: Payton Staub <[email protected]> Co-authored-by: Payton Staub <[email protected]> Co-authored-by: Shreya Pandit <[email protected]> Co-authored-by: Ahsan Khan <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]>

BasilBeirouti force-pushed the dev branch from 0e4fd55 to 26b984a Compare February 23, 2022 02:20

Payton Staub added 2 commits March 2, 2022 10:34

fix: Remove sagemaker_job_name from hyperparameters in TrainingStep

8118a81

Fix unit tests

6ff0c7a

staubhp force-pushed the fix-training-step-caching branch from 298f965 to 6ff0c7a Compare March 2, 2022 18:34

black-format

dbddcc2

jerrypeng7773 reviewed Mar 2, 2022

View reviewed changes

qidewenwhen approved these changes Mar 2, 2022

View reviewed changes

Add property check

2dfde22

shreyapandit reviewed Mar 3, 2022

View reviewed changes

tests/unit/sagemaker/workflow/test_step_collections.py Show resolved Hide resolved

shreyapandit suggested changes Mar 3, 2022

View reviewed changes

shreyapandit added the type: question label Mar 3, 2022

jerrypeng7773 approved these changes Mar 3, 2022

View reviewed changes

shreyapandit approved these changes Mar 3, 2022

View reviewed changes

shreyapandit merged commit 20df3d7 into aws:dev Mar 3, 2022

shreyapandit removed the type: question label Mar 3, 2022

shreyapandit pushed a commit that referenced this pull request Mar 4, 2022

fix: Remove sagemaker_job_name from hyperparameters in TrainingStep (#…

067cac2

…2950) Co-authored-by: Payton Staub <[email protected]>

jerrypeng7773 pushed a commit to jerrypeng7773/sagemaker-python-sdk that referenced this pull request Apr 19, 2022

fix: Remove sagemaker_job_name from hyperparameters in TrainingStep (a…

5ea9f21

…ws#2950) Co-authored-by: Payton Staub <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Remove sagemaker_job_name from hyperparameters in TrainingStep #2950

fix: Remove sagemaker_job_name from hyperparameters in TrainingStep #2950

staubhp commented Feb 22, 2022

codecov-commenter commented Feb 23, 2022 •

edited

Loading

sagemaker-bot commented Feb 23, 2022

sagemaker-bot commented Feb 23, 2022

sagemaker-bot commented Feb 23, 2022

sagemaker-bot commented Mar 2, 2022

sagemaker-bot commented Mar 2, 2022

sagemaker-bot commented Mar 2, 2022

sagemaker-bot commented Mar 2, 2022

sagemaker-bot commented Mar 2, 2022

jerrypeng7773 Mar 2, 2022

staubhp Mar 3, 2022

sagemaker-bot commented Mar 2, 2022

sagemaker-bot commented Mar 2, 2022

sagemaker-bot commented Mar 2, 2022

sagemaker-bot commented Mar 3, 2022

sagemaker-bot commented Mar 3, 2022

sagemaker-bot commented Mar 3, 2022

shreyapandit left a comment

fix: Remove sagemaker_job_name from hyperparameters in TrainingStep #2950

fix: Remove sagemaker_job_name from hyperparameters in TrainingStep #2950

Conversation

staubhp commented Feb 22, 2022

Merge Checklist

General

Tests

codecov-commenter commented Feb 23, 2022 • edited Loading

Codecov Report

sagemaker-bot commented Feb 23, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 23, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 23, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 2, 2022

AWS CodeBuild CI Report

jerrypeng7773 Mar 2, 2022

Choose a reason for hiding this comment

staubhp Mar 3, 2022

Choose a reason for hiding this comment

sagemaker-bot commented Mar 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2022

AWS CodeBuild CI Report

shreyapandit left a comment

Choose a reason for hiding this comment

codecov-commenter commented Feb 23, 2022 •

edited

Loading