change: Add label_headers option for Clarify ModelExplainabilityMonitor #2707

xgchena · 2021-10-15T02:41:43Z

Description of changes:

For Clarify explainability job, the label_headers option is used to beautify the analysis report by replacing placeholders like "label0". The option has been added to SageMakerClarifyProcessor API by PR #2446, this commit adds the same option to ModelExplainabilityMonitor.

Testing done:

Unit Test passed
Integ Test passed and the label name in analysis reports is replaced as expected.

For tracking: https://tiny.amazon.com/1i1p4eh9w

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2021-10-15T02:56:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: e40a058
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-15T03:34:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: e40a058
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

xgchena · 2021-10-15T03:40:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests

Commit ID: e40a058

Result: FAILED

Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Error message: "FAILED tests/integ/test_lda.py::test_lda - sagemaker.exceptions.UnexpectedStatusException"

The failure is not caused by the PR. The PR only touched Clarify code and "tests/integ/test_clarify_model_monitor.py" has passed.

sagemaker-bot · 2021-10-15T03:45:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: e40a058
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

tests/integ/test_clarify_model_monitor.py

larroy

minor comment, LGTM

xgchena · 2021-10-30T04:02:45Z

Rebased to resolve conflicts.

sagemaker-bot · 2021-10-30T04:16:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: dd954b1
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-30T04:59:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: dd954b1
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-30T05:12:48Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: dd954b1
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

xgchena · 2021-10-30T05:25:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests

Commit ID: dd954b1

Result: FAILED

Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

The log is incomplete/not showing the failure.

xgchena · 2021-10-30T05:28:23Z

No change, force push to rebuild and hope this time the slow tests either pass or show full build log.

sagemaker-bot · 2021-10-30T05:41:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 99d54be
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-30T06:31:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 99d54be
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

xgchena · 2021-11-10T19:15:50Z

Rebase and force push to trigger checks.

codecov-commenter · 2021-11-10T19:28:05Z

Codecov Report

Merging #2707 (dc987b9) into dev (87c1d2c) will decrease coverage by 0.00%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##              dev    #2707      +/-   ##
==========================================
- Coverage   88.67%   88.67%   -0.01%     
==========================================
  Files         173      173              
  Lines       15429    15441      +12     
==========================================
+ Hits        13682    13692      +10     
- Misses       1747     1749       +2

Impacted Files	Coverage Δ
...agemaker/model_monitor/clarify_model_monitoring.py	`95.48% <84.61%> (-0.44%)`	⬇️
src/sagemaker/clarify.py	`92.40% <100.00%> (+0.03%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 87c1d2c...dc987b9. Read the comment docs.

sagemaker-bot · 2021-11-10T19:31:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 5f6e797
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

xgchena · 2021-12-09T19:30:20Z

Changed target branch to dev which is now the default branch.
Rebased from latest dev branch code.
The change was reviewed and approved by Pedro, after that there are only rebases and minor changes to resolve conflicts.

sagemaker-bot · 2021-12-09T19:42:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 7b1fb07
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

#2790)

…ries (#2755) Co-authored-by: Payton Staub <[email protected]> Co-authored-by: Ahsan Khan <[email protected]>

sagemaker-bot · 2021-12-15T21:17:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: df21794
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-12-15T22:13:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: df21794
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Co-authored-by: Shreya Pandit <[email protected]>

xgchena · 2021-12-17T22:40:05Z

Shreya granted me access to the CodeBuild account to view build logs.

Below are the failure reasons and I can confirm they are not caused by changes of this PR.

AWS CodeBuild us-west-2 (sagemaker-python-sdk-notebook-tests)

One notebook failed,

--------------------------------------------------------------------------------
SUCCEEDED   11
FAILED  1
ERROR   0
SKIPPED 0
CANCELLED   0
TIMED_OUT   0
--------------------------------------------------------------------------------
...
FAILED  ./amazon-sagemaker-examples/sagemaker-pipelines/tabular/abalone_build_train_deploy/sagemaker-pipelines-preprocess-train-evaluate-batch-transform.ipynb  PT31M34.794S
...

AWS CodeBuild us-west-2 (sagemaker-python-sdk-pr)

Four test cases failed,

=========================== short test summary info ============================
FAILED tests/integ/test_kmeans.py::test_kmeans - botocore.exceptions.ClientEr...
FAILED tests/integ/test_multidatamodel.py::test_multi_data_model_deploy_pretrained_models
FAILED tests/integ/test_multidatamodel.py::test_multi_data_model_deploy_pretrained_models_update_endpoint
FAILED tests/integ/test_multidatamodel.py::test_multi_data_model_deploy_trained_model_from_framework_estimator[1.8.0-py37]
= 4 failed, 282 passed, 24 skipped, 23448 warnings, 82 rerun in 5607.57s (1:33:27) =

AWS CodeBuild us-west-2 (sagemaker-python-sdk-slow-tests)

This test failed due to "BUILD_TIMED_OUT: Build has timed out. "

Co-authored-by: Basil Beirouti <[email protected]> Co-authored-by: Payton Staub <[email protected]> Co-authored-by: Ahsan Khan <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]> Co-authored-by: Basil Beirouti <[email protected]> Co-authored-by: Payton Staub <[email protected]> Co-authored-by: Shreya Pandit <[email protected]>

sagemaker-bot · 2021-12-18T00:08:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 6d6eea5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-12-18T00:24:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: eb8dbd7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-12-18T00:25:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 6d6eea5
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/clarify.py

src/sagemaker/model_monitor/clarify_model_monitoring.py

The option has been added to SageMakerClarifyProcessor API by PR 2446, this commit adds the same option to ModelExplainabilityMonitor.

sagemaker-bot · 2021-12-20T18:16:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 940b1f1
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-12-30T00:50:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 999149e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-12-30T01:24:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 999149e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

* update smdmp change log, archive api doc for 1.4.0 and 1.5.0 * add no-index flags * finish api doc archive * fix: Set ProcessingStep upload locations deterministically to avoid c… (#2790) * fix: Prevent repack_model script from referencing nonexistent directories (#2755) Co-authored-by: Payton Staub <[email protected]> Co-authored-by: Ahsan Khan <[email protected]> * fix: S3Input - add support for instance attributes (#2754) * fix: typos and broken link (#2765) Co-authored-by: Shreya Pandit <[email protected]> * add all api docs * add appendix, fix links * structural changes, fix links * incorporate feedback * prepare release v2.72.1 * update development version to v2.72.2.dev0 Co-authored-by: Payton Staub <[email protected]> Co-authored-by: Payton Staub <[email protected]> Co-authored-by: Ahsan Khan <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]> Co-authored-by: Mohamed Ali Jamaoui <[email protected]> Co-authored-by: Shreya Pandit <[email protected]> Co-authored-by: ci <ci> Co-authored-by: Jeniya Tabassum <[email protected]>

)

sagemaker-bot · 2022-01-04T03:01:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 3674e08
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-06T19:46:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: dc987b9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-06T20:25:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: dc987b9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-06T20:27:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: dc987b9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-06T21:01:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: dc987b9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-06T21:11:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: dc987b9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…or (aws#2707) Co-authored-by: Payton Staub <[email protected]> Co-authored-by: Ahsan Khan <[email protected]> Co-authored-by: Shreya Pandit <[email protected]> Co-authored-by: Basil Beirouti <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]> Co-authored-by: Basil Beirouti <[email protected]> Co-authored-by: Payton Staub <[email protected]> Co-authored-by: Mohamed Ali Jamaoui <[email protected]> Co-authored-by: ci <ci> Co-authored-by: Jeniya Tabassum <[email protected]> Co-authored-by: sreedes <[email protected]> Co-authored-by: Navin Soni <[email protected]> Co-authored-by: Miyoung <[email protected]>

larroy reviewed Oct 15, 2021

View reviewed changes

tests/integ/test_clarify_model_monitor.py Show resolved Hide resolved

larroy previously approved these changes Oct 15, 2021

View reviewed changes

xgchena dismissed larroy’s stale review via dd954b1 October 30, 2021 04:02

xgchena changed the base branch from master to dev December 9, 2021 19:27

staubhp and others added 3 commits December 15, 2021 10:45

fix: Set ProcessingStep upload locations deterministically to avoid c… (

dde8d00

#2790)

fix: Prevent repack_model script from referencing nonexistent directo…

0f72907

…ries (#2755) Co-authored-by: Payton Staub <[email protected]> Co-authored-by: Ahsan Khan <[email protected]>

fix: S3Input - add support for instance attributes (#2754)

0bae071

shreyapandit force-pushed the dev branch from 2f7c40a to 0bae071 Compare December 15, 2021 19:38

fix: typos and broken link (#2765)

17fe93e

Co-authored-by: Shreya Pandit <[email protected]>

navinsoni reviewed Dec 18, 2021

View reviewed changes

src/sagemaker/clarify.py Outdated Show resolved Hide resolved

src/sagemaker/clarify.py Outdated Show resolved Hide resolved

src/sagemaker/model_monitor/clarify_model_monitoring.py Outdated Show resolved Hide resolved

change: Add label_headers option for Clarify ModelExplainabilityMonitor

940b1f1

The option has been added to SageMakerClarifyProcessor API by PR 2446, this commit adds the same option to ModelExplainabilityMonitor.

navinsoni self-requested a review December 22, 2021 22:27

navinsoni approved these changes Dec 22, 2021

View reviewed changes

navinns and others added 3 commits December 29, 2021 16:21

fix: Add ContentType in test_auto_ml_describe

185b4b0

fix: Re-deploy static integ test endpoint if it is not found

5ec2ff4

Merge branch 'dev' into mm-explainability-label-headers

999149e

mchoi8739 and others added 2 commits December 30, 2021 07:30

fix: fix kmeans test deletion sequence, increment lineage statics (#2815

6319bce

)

mufaddal-rohawala requested a review from BasilBeirouti January 3, 2022 21:08

mufaddal-rohawala added 2 commits January 3, 2022 15:42

fix: Increment static lineage pipeline (#2817)

beeabbd

Merge branch 'dev' into mm-explainability-label-headers

3674e08

mufaddal-rohawala force-pushed the dev branch from beeabbd to b026769 Compare January 4, 2022 17:09

BasilBeirouti approved these changes Jan 4, 2022

View reviewed changes

xiaoyi-cheng approved these changes Jan 6, 2022

View reviewed changes

Merge branch 'dev' into mm-explainability-label-headers

dc987b9

ahsan-z-khan merged commit 496b259 into aws:dev Jan 7, 2022

change: Add label_headers option for Clarify ModelExplainabilityMonitor #2707

change: Add label_headers option for Clarify ModelExplainabilityMonitor #2707

Conversation

xgchena commented Oct 15, 2021 • edited Loading

Merge Checklist

General

Tests

sagemaker-bot commented Oct 15, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 15, 2021

AWS CodeBuild CI Report

xgchena commented Oct 15, 2021 • edited Loading

AWS CodeBuild CI Report

sagemaker-bot commented Oct 15, 2021

AWS CodeBuild CI Report

larroy left a comment

Choose a reason for hiding this comment

xgchena commented Oct 30, 2021

sagemaker-bot commented Oct 30, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 30, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 30, 2021

AWS CodeBuild CI Report

xgchena commented Oct 30, 2021

AWS CodeBuild CI Report

xgchena commented Oct 30, 2021

sagemaker-bot commented Oct 30, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 30, 2021

AWS CodeBuild CI Report

xgchena commented Nov 10, 2021

codecov-commenter commented Nov 10, 2021 • edited Loading

Codecov Report

sagemaker-bot commented Nov 10, 2021

AWS CodeBuild CI Report

xgchena commented Dec 9, 2021

sagemaker-bot commented Dec 9, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Dec 15, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Dec 15, 2021

AWS CodeBuild CI Report

xgchena commented Dec 17, 2021

sagemaker-bot commented Dec 18, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Dec 18, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Dec 18, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Dec 20, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Dec 30, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Dec 30, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jan 4, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

xgchena commented Oct 15, 2021 •

edited

Loading

xgchena commented Oct 15, 2021 •

edited

Loading

codecov-commenter commented Nov 10, 2021 •

edited

Loading