change: make model_config optional when predicted labels are provided for bias detection #3596

oyangz · 2023-01-17T11:19:11Z

Issue #, if available:
https://issues.amazon.com/issues/RAI-3961

Description of changes:

Made model_config optional when predicted labels are provided for bias detection
Added unit tests and integration test for post training bias with predicted labels
Updated parameter descriptions in run_bias and run_post_training_bias

Testing done:

manually tested changes locally by running bias detection in jupyter notebook, verified run_bias and run_post_training_bias can be called without model_config when predicted labels are provided.
unit tests and clarify integration tests passed

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2023-01-17T11:25:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 0ad0232
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-17T11:59:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 0ad0232
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2023-01-17T12:00:00Z

Codecov Report

Merging #3596 (975a5cb) into master (1fa2377) will decrease coverage by 0.36%.
The diff coverage is 87.74%.

@@            Coverage Diff             @@
##           master    #3596      +/-   ##
==========================================
- Coverage   89.17%   88.82%   -0.36%     
==========================================
  Files         204      227      +23     
  Lines       18979    22312    +3333     
==========================================
+ Hits        16924    19818    +2894     
- Misses       2055     2494     +439

Impacted Files	Coverage Δ
src/sagemaker/dataset_definition/inputs.py	`100.00% <ø> (ø)`
src/sagemaker/debugger/framework_profile.py	`100.00% <ø> (ø)`
src/sagemaker/debugger/metrics_config.py	`100.00% <ø> (ø)`
src/sagemaker/huggingface/model.py	`89.87% <ø> (ø)`
src/sagemaker/lineage/_utils.py	`100.00% <ø> (+12.50%)`	⬆️
src/sagemaker/model_card/model_card.py	`93.04% <ø> (ø)`
src/sagemaker/model_card/schema_constraints.py	`100.00% <ø> (ø)`
src/sagemaker/model_monitor/__init__.py	`100.00% <ø> (ø)`
...agemaker/model_monitor/clarify_model_monitoring.py	`89.84% <ø> (-5.09%)`	⬇️
src/sagemaker/model_monitor/dataset_format.py	`72.72% <ø> (-2.28%)`	⬇️
... and 110 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

sagemaker-bot · 2023-01-17T12:00:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 0ad0232
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-17T12:02:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 0ad0232
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-17T12:21:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 0ad0232
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

larroy

LGTM

sagemaker-bot · 2023-01-19T23:52:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 2836510
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-19T23:55:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 2836510
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T00:04:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 2836510
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T00:26:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 2836510
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T00:31:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 2836510
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T21:59:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 25ce88a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T22:00:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 25ce88a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T22:02:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 25ce88a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T22:13:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 25ce88a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T22:25:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 25ce88a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

vikas0203

/bot run slow-tests

sagemaker-bot · 2023-01-21T01:53:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 25ce88a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-24T19:36:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 41f18c3
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-24T20:10:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 41f18c3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-24T20:11:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 41f18c3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-24T20:21:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 41f18c3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-26T08:43:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 39b2e65
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

vikas0203

/bot run notebook-tests,pr,slow-tests

sagemaker-bot · 2023-02-02T17:49:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 39b2e65
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-02T18:26:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 39b2e65
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-02T18:50:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 39b2e65
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-07T01:20:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: b472b3f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-07T01:55:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: b472b3f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-07T01:59:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: b472b3f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-07T02:08:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: b472b3f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-07T02:21:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: b472b3f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-07T19:36:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: f68d41e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-07T19:44:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: f68d41e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-07T20:07:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: f68d41e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-07T20:24:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: f68d41e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-07T21:48:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: f68d41e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-07T23:05:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 975a5cb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-07T23:08:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 975a5cb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-07T23:16:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 975a5cb
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-07T23:21:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 975a5cb
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-07T23:33:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 975a5cb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

vikas0203

/bot run slow-tests,pr

sagemaker-bot · 2023-02-08T19:22:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 975a5cb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

vikas0203

/bot run pr

sagemaker-bot · 2023-02-08T20:06:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 975a5cb
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

… for bias detection (aws#3596)

oyangz marked this pull request as ready for review January 17, 2023 21:40

oyangz requested a review from a team as a code owner January 17, 2023 21:40

oyangz requested review from vikas0203 and removed request for a team January 17, 2023 21:40

larroy approved these changes Jan 17, 2023

View reviewed changes

claytonparnell assigned vikas0203 Jan 18, 2023

xiaoyi-cheng approved these changes Jan 20, 2023

View reviewed changes

vikas0203 reviewed Jan 20, 2023

View reviewed changes

oyangz force-pushed the model_config branch from 25ce88a to 41f18c3 Compare January 24, 2023 19:28

vikas0203 reviewed Feb 2, 2023

View reviewed changes

Merge branch 'aws:master' into model_config

b472b3f

Merge branch 'aws:master' into model_config

f68d41e

Merge branch 'aws:master' into model_config

975a5cb

vikas0203 reviewed Feb 8, 2023

View reviewed changes

vikas0203 merged commit 1c9dd1d into aws:master Feb 10, 2023

JoseJuan98 pushed a commit to JoseJuan98/sagemaker-python-sdk that referenced this pull request Mar 4, 2023

change: make model_config optional when predicted labels are provided…

62601d3

… for bias detection (aws#3596)

nmadan pushed a commit to nmadan/sagemaker-python-sdk that referenced this pull request Apr 18, 2023

change: make model_config optional when predicted labels are provided…

1c28c53

… for bias detection (aws#3596)

change: make model_config optional when predicted labels are provided for bias detection #3596

change: make model_config optional when predicted labels are provided for bias detection #3596

Conversation

oyangz commented Jan 17, 2023

Merge Checklist

General

Tests

sagemaker-bot commented Jan 17, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 17, 2023

AWS CodeBuild CI Report

codecov-commenter commented Jan 17, 2023 • edited Loading

Codecov Report

sagemaker-bot commented Jan 17, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 17, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 17, 2023

AWS CodeBuild CI Report

larroy left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Jan 19, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 19, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

vikas0203 left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Jan 21, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 24, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 24, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 24, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 24, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 26, 2023

AWS CodeBuild CI Report

vikas0203 left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Feb 2, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 2, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 2, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 7, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 7, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 7, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 7, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 7, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 7, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 7, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 7, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 7, 2023

codecov-commenter commented Jan 17, 2023 •

edited

Loading