Added bias_and_explainability to SageMakerClarifyProcessor #3282

dosatos · 2022-08-05T14:10:06Z

Issue #, if available:

Description of changes:
Added a public method to run both Bias and Explainability in a single job processing

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

keerthanvasist · 2022-08-05T18:20:09Z

src/sagemaker/clarify.py

+            experiment_config,
+        )
+
+    def run_bias_and_explainability(self):


Can we see if we can override the existing run() method for this purpose?

I think that for run() we have different input schema

def run( self, inputs: Optional[List["ProcessingInput"]] = None, outputs: Optional[List["ProcessingOutput"]] = None, arguments: Optional[List[Union[str, PipelineVariable]]] = None, wait: bool = True, logs: bool = True, job_name: Optional[str] = None, experiment_config: Optional[Dict[str, str]] = None, kms_key: Optional[str] = None, ):

Signature of method 'SageMakerClarifyProcessor.run()' does not match signature of the base method in class 'Processor'

We could potentially expose the config generator, but I do not think that's improves customer experience. Let's leave everything as-is - the intetrace/arguments that the customer already knows

dosatos · 2022-08-08T15:50:10Z

I will apply formatting after the PR #3271 gets merged - it partially improves the style already

sagemaker-bot · 2022-08-11T17:16:48Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 295fe83
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-11T17:18:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 295fe83
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-11T18:00:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 295fe83
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-11T18:05:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 295fe83
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-11T18:21:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 295fe83
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T10:54:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: d54f2bf
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T10:59:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 5dcaca3
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T11:44:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 5dcaca3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T12:01:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 5dcaca3
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T12:04:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 5dcaca3
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T12:37:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 4fa6f98
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T13:07:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 67708bf
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T13:25:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 6f82a43
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T13:26:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6f82a43
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T13:37:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: b2e72d6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T13:39:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 48131a5
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T13:45:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: fb6f568
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T13:47:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: fb6f568
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T14:05:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 76eb325
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T14:07:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 76eb325
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T14:15:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 6fc02f4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T14:27:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 6fc02f4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T14:42:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6fc02f4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2022-08-12T14:44:00Z

Codecov Report

Merging #3282 (2dc3867) into master (38b4916) will decrease coverage by 0.35%.
The diff coverage is 96.03%.

@@            Coverage Diff             @@
##           master    #3282      +/-   ##
==========================================
- Coverage   89.40%   89.04%   -0.36%     
==========================================
  Files         848      203     -645     
  Lines       72563    18345   -54218     
==========================================
- Hits        64877    16336   -48541     
+ Misses       7686     2009    -5677

Impacted Files	Coverage Δ
src/sagemaker/jumpstart/types.py	`94.37% <0.00%> (ø)`
src/sagemaker/processing.py	`94.33% <ø> (ø)`
...agemaker/serverless/serverless_inference_config.py	`100.00% <ø> (ø)`
src/sagemaker/transformer.py	`94.47% <ø> (ø)`
src/sagemaker/workflow/step_collections.py	`93.87% <ø> (ø)`
src/sagemaker/job.py	`90.00% <75.00%> (-0.55%)`	⬇️
src/sagemaker/tensorflow/estimator.py	`92.18% <75.00%> (-0.56%)`	⬇️
src/sagemaker/estimator.py	`89.24% <76.47%> (-0.35%)`	⬇️
src/sagemaker/chainer/model.py	`96.49% <88.88%> (-1.07%)`	⬇️
src/sagemaker/mxnet/model.py	`98.41% <90.00%> (+2.04%)`	⬆️
... and 705 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

sagemaker-bot · 2022-08-12T15:00:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 6fc02f4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T15:19:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 6fc02f4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

larroy

I like the change in general, minor issues.

src/sagemaker/clarify.py

larroy · 2022-08-15T20:40:51Z

src/sagemaker/clarify.py

+        model_predicted_label_config: ModelPredictedLabelConfig,
+        explainability_config: Union[ExplainabilityConfig, List[ExplainabilityConfig]],
+        bias_config: BiasConfig,
+        pre_training_methods: Union[str, List[str]] = "all",


I'm not a big fan of these Union types, I see the intention, but is it worth it to save the user typing [] ? I would prefer if it's a plain list and we check that is not empty. Open to discuss about this and be convinced of the opposite though.

Are you saying that, instead of making the user to pass "all", we can make it ["all"]?
Actually, I did it this way to be consistent with the current implementation - i.e. with how we expose methods yet.

I mean explainability_config: Union[ExplainabilityConfig, List[ExplainabilityConfig]],

So it was initially only ExplainabilityConfig, and we only had SHAPConfig. But then we added PDPConfig, and we wanted to let customers be allowed to choose both of them in a single job. So, we made into List[ExplainabilityConfig] but also kept the original single ExplainabilityConfig as well to maintain backward compatibility.

I see this was like that and you just annotated the same type, thanks for catching up with this.

shreyapandit · 2022-08-15T23:52:43Z

I have restarted PR and Local Mode tests, notebook failures are expected and I can override them. Please have 1 approval from @larroy or @keerthanvasist for us to be able to merge

sagemaker-bot · 2022-08-16T00:33:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 6fc02f4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-16T10:16:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 2dc3867
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-16T10:39:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 2dc3867
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-16T10:58:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 2dc3867
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-16T10:59:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 2dc3867
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-16T12:38:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 2dc3867
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

larroy · 2022-08-16T19:01:57Z

src/sagemaker/clarify.py

+        model_predicted_label_config: ModelPredictedLabelConfig,
+        explainability_config: Union[ExplainabilityConfig, List[ExplainabilityConfig]],
+        bias_config: BiasConfig,
+        pre_training_methods: Union[str, List[str]] = "all",


I see this was like that and you just annotated the same type, thanks for catching up with this.

keerthanvasist · 2022-08-16T19:43:22Z

src/sagemaker/clarify.py

+            analysis_config, model_config, model_predicted_label_config
+        )
+        return analysis_config
+
    @classmethod
    def explainability(
        cls,


Just to maintain consistency, can you use self instead of cls?

Thanks for the comment, Keerthan.self implies creation of an instance, cls does not require an instance
This should also be driven by pep8.

dosatos · 2022-08-18T09:13:24Z

The changes in this PR were also present in this one: #3296

Thus, cancelling the PR

dosatos changed the title ~~Clarify/bias and explainability api~~ Added bias_and_explainability to SageMakerClarifyProcessor Aug 5, 2022

keerthanvasist reviewed Aug 5, 2022

View reviewed changes

dosatos force-pushed the clarify/bias_and_explainability_api branch 3 times, most recently from 49a596e to 8eb4b01 Compare August 8, 2022 15:49

dosatos force-pushed the clarify/bias_and_explainability_api branch from 8eb4b01 to 2bb714f Compare August 8, 2022 15:51

dosatos force-pushed the clarify/bias_and_explainability_api branch 2 times, most recently from 3300f60 to 121ca9d Compare August 12, 2022 10:48

dosatos force-pushed the clarify/bias_and_explainability_api branch from d54f2bf to 5dcaca3 Compare August 12, 2022 10:55

dosatos force-pushed the clarify/bias_and_explainability_api branch from 5dcaca3 to 4fa6f98 Compare August 12, 2022 12:33

dosatos force-pushed the clarify/bias_and_explainability_api branch from 4fa6f98 to 67708bf Compare August 12, 2022 13:02

dosatos force-pushed the clarify/bias_and_explainability_api branch from 6f82a43 to b2e72d6 Compare August 12, 2022 13:32

dosatos force-pushed the clarify/bias_and_explainability_api branch from b2e72d6 to 48131a5 Compare August 12, 2022 13:37

dosatos force-pushed the clarify/bias_and_explainability_api branch 3 times, most recently from b323930 to 76eb325 Compare August 12, 2022 14:00

dosatos force-pushed the clarify/bias_and_explainability_api branch from 76eb325 to 6fc02f4 Compare August 12, 2022 14:13

larroy suggested changes Aug 15, 2022

View reviewed changes

dosatos requested review from larroy and keerthanvasist August 15, 2022 21:19

added run_bias_and_explainability method

2dc3867

dosatos force-pushed the clarify/bias_and_explainability_api branch from 6fc02f4 to 2dc3867 Compare August 16, 2022 09:46

larroy approved these changes Aug 16, 2022

View reviewed changes

keerthanvasist approved these changes Aug 16, 2022

View reviewed changes

dosatos closed this Aug 18, 2022

Added bias_and_explainability to SageMakerClarifyProcessor #3282

Added bias_and_explainability to SageMakerClarifyProcessor #3282

Uh oh!

Conversation

dosatos commented Aug 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Checklist

General

Tests

Uh oh!

keerthanvasist Aug 5, 2022

Choose a reason for hiding this comment

Uh oh!

dosatos Aug 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dosatos commented Aug 8, 2022

Uh oh!

sagemaker-bot commented Aug 11, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 11, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 11, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 11, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 11, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

dosatos commented Aug 5, 2022 •

edited

Loading

dosatos Aug 8, 2022 •

edited

Loading

codecov-commenter commented Aug 12, 2022 •

edited

Loading

keerthanvasist Aug 16, 2022 •

edited

Loading