Revert #2560 #2564

verdimrc · 2021-08-07T07:38:04Z

Issue #, if available:

Description of changes:

Framework processor uses a python3 bootstrap script
Re-enable the new SKLearn processor

Testing done: unit, integ.

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2021-08-07T07:49:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 0ecb9a0
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-08-07T08:12:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 0ecb9a0
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-08-07T08:14:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 0ecb9a0
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-08-07T08:19:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 0ecb9a0
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-08-07T08:25:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 0ecb9a0
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-08-07T08:33:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 4a562bd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-08-07T08:56:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 4a562bd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-08-07T08:59:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 4a562bd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-08-07T09:01:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 4a562bd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-08-07T09:07:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 4a562bd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

ahsan-z-khan · 2021-08-07T17:53:40Z

@verdimrc Can I have the zip file to test?

verdimrc · 2021-08-08T02:02:30Z

Sample source dir. In the processor, use code="processing.py".

sample-sourcedir.zip

sample-sourcedir-no-req.zip

ahsan-z-khan · 2021-08-08T19:49:07Z

tests/integ/test_local_mode.py

@@ -349,12 +349,12 @@ def test_local_processing_sklearn(sagemaker_local_session_no_local_code, sklearn

    job_description = sklearn_processor.latest_job.describe()

-    assert len(job_description["ProcessingInputs"]) == 2
+    assert len(job_description["ProcessingInputs"]) == 3


I’m trying to understand why there is now one more ProcessingInputs?

@verdimrc @athewsey any updates?

Hi, apology, my reply went to the PR comment instead of this area.

The test for the original sklearn processor creates these 2x input channels:

/opt/ml/processing/inputs => input data

/opt/ml/processing/input/code => to place dummy_script.py which is the container entrypoint.

The new sklearn processor creates these 3x input channels:

/opt/ml/processing/inputs => input data

/opt/ml/processing/input/code => to place sourcedir.tar.gz which contains dummy_script.py

/opt/ml/processing/input/entrypoint => to place bootstrap logic runproc.py which is the container entrypoint. See this.

verdimrc · 2021-08-09T03:37:11Z

hi Ahsan, the extra input is for the sourcedir.tar.gz. The original code input (from Script processor) is used for the bootstrapping script. Regards, Verdi

…

On Mon, 9 Aug 2021, 3:49 am Ahsan Khan, ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In tests/integ/test_local_mode.py <#2564 (comment)> : > @@ -349,12 +349,12 @@ def test_local_processing_sklearn(sagemaker_local_session_no_local_code, sklearn job_description = sklearn_processor.latest_job.describe() - assert len(job_description["ProcessingInputs"]) == 2 + assert len(job_description["ProcessingInputs"]) == 3 I’m trying to understand why there is now one more ProcessingInputs? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2564 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAR3PLI5TDG2SUPIYTH7HSDT33NUFANCNFSM5BXEYGOQ> .

sagemaker-bot · 2021-08-16T04:36:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 30417db
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-08-17T19:07:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 4e160c7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-08-20T21:51:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 78eba90
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-09-20T19:45:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a904fff
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

DaniloTommasinaTR · 2021-09-21T14:55:04Z

Hi guys, any news on this one? We are really keen to make use of these changes.
Thanks.

verdimrc · 2021-09-22T08:40:38Z

This PR fixes the only symptom. However, the root-cause is still there: ProcessingStep calls the processor’s private API _normalize_args(), which essentially short-circuits the framework processor.

This will cause two consequences:

(1) pipeline still doesn’t support requirements.txt.
(2) potential maintenance difficulty as ProcessingStep replicates incomplete mechanics from FrameworkProcessor.

A new PR #2633 has been opened to directly addresses the root cause.

verdimrc · 2021-10-06T13:38:13Z

(Pushed to draft in favour of #2664)

The base branch was changed.

ahsan-z-khan · 2022-01-18T19:15:35Z

@verdimrc Do we need this PR after #2664 ?

verdimrc · 2022-01-18T23:44:38Z

Hi @ahsan-z-khan, this PR is not needed anymore.

Verdi March added 3 commits August 7, 2021 14:14

Change bash boostrap to python3 bootstrap

39158b8

Re-enable the new SKLearnProcessor

b2f4cb5

Fix runproc.py generation

0ecb9a0

Fixed unit tests for huggingface processor

4a562bd

ahsan-z-khan reviewed Aug 8, 2021

View reviewed changes

ahsan-z-khan added the component: processing Relates to the SageMaker Processing Platform label Aug 10, 2021

Merge branch 'master' into framework-processor-python3

30417db

Merge branch 'master' into framework-processor-python3

4e160c7

Merge branch 'master' into framework-processor-python3

78eba90

ahsan-z-khan previously approved these changes Aug 20, 2021

View reviewed changes

verdimrc mentioned this pull request Aug 26, 2021

feature: processors that support multiple Python files, requirements.txt, and dependencies. #2251

Merged

7 tasks

ahsan-z-khan added 2 commits September 9, 2021 13:13

Merge branch 'master' into framework-processor-python3

547425c

Merge branch 'master' into framework-processor-python3

b52ef18

athewsey mentioned this pull request Sep 14, 2021

Restore SKLearn FrameworkProcessor via _normalize_args #2633

Closed

7 tasks

Merge branch 'master' into framework-processor-python3

a904fff

Merge branch 'master' into framework-processor-python3

88bf420

verdimrc marked this pull request as draft October 6, 2021 13:37

ahsan-z-khan mentioned this pull request Oct 21, 2021

Fix FrameworkProcessor for SageMaker Pipelines #2664

Closed

9 tasks

shreyapandit changed the base branch from master to dev November 5, 2021 17:27

mufaddal-rohawala force-pushed the dev branch from e0b9d38 to 2beb91e Compare December 6, 2021 18:41

mufaddal-rohawala force-pushed the dev branch from beeabbd to b026769 Compare January 4, 2022 17:09

shreyapandit force-pushed the dev branch from 1c039c8 to 7206b9e Compare January 10, 2022 23:05

shreyapandit force-pushed the dev branch from 903205c to b796a00 Compare January 18, 2022 20:09

ahsan-z-khan closed this Jan 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revert #2560 #2564

Revert #2560 #2564

verdimrc commented Aug 7, 2021

sagemaker-bot commented Aug 7, 2021

sagemaker-bot commented Aug 7, 2021

sagemaker-bot commented Aug 7, 2021

sagemaker-bot commented Aug 7, 2021

sagemaker-bot commented Aug 7, 2021

sagemaker-bot commented Aug 7, 2021

sagemaker-bot commented Aug 7, 2021

sagemaker-bot commented Aug 7, 2021

sagemaker-bot commented Aug 7, 2021

sagemaker-bot commented Aug 7, 2021

ahsan-z-khan commented Aug 7, 2021

verdimrc commented Aug 8, 2021

ahsan-z-khan Aug 8, 2021

shreyapandit Aug 16, 2021

verdimrc Aug 17, 2021 •

edited

Loading

verdimrc commented Aug 9, 2021 via email

sagemaker-bot commented Aug 16, 2021

sagemaker-bot commented Aug 17, 2021

sagemaker-bot commented Aug 20, 2021

sagemaker-bot commented Sep 20, 2021

DaniloTommasinaTR commented Sep 21, 2021

verdimrc commented Sep 22, 2021 •

edited

Loading

verdimrc commented Oct 6, 2021

ahsan-z-khan commented Jan 18, 2022

verdimrc commented Jan 18, 2022

Revert #2560 #2564

Revert #2560 #2564

Conversation

verdimrc commented Aug 7, 2021

Merge Checklist

General

Tests

sagemaker-bot commented Aug 7, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Aug 7, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Aug 7, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Aug 7, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Aug 7, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Aug 7, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Aug 7, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Aug 7, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Aug 7, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Aug 7, 2021

AWS CodeBuild CI Report

ahsan-z-khan commented Aug 7, 2021

verdimrc commented Aug 8, 2021

ahsan-z-khan Aug 8, 2021

Choose a reason for hiding this comment

shreyapandit Aug 16, 2021

Choose a reason for hiding this comment

verdimrc Aug 17, 2021 • edited Loading

Choose a reason for hiding this comment

verdimrc commented Aug 9, 2021 via email

sagemaker-bot commented Aug 16, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Aug 17, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Aug 20, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Sep 20, 2021

AWS CodeBuild CI Report

DaniloTommasinaTR commented Sep 21, 2021

verdimrc commented Sep 22, 2021 • edited Loading

verdimrc commented Oct 6, 2021

ahsan-z-khan commented Jan 18, 2022

verdimrc commented Jan 18, 2022

verdimrc Aug 17, 2021 •

edited

Loading

verdimrc commented Sep 22, 2021 •

edited

Loading