Skip to content

feature: Add support for SageMaker lineage queries context #2830

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 19, 2022

Conversation

yzhu0
Copy link
Contributor

@yzhu0 yzhu0 commented Jan 6, 2022

Issue #, if available:

Description of changes:
Add support for SageMaker lineage queries context

  • Return Pipeline Execution in ModelPackageGroup

Testing done:
Integration test
tox -e py36 -- tests/integ/sagemaker/lineage/test_model_package_group_context.py -vv

tests/integ/sagemaker/lineage/test_model_package_group_context.py::test_pipeline_execution_arn PASSED [100%]

tox -e py36 -- tests/integ/sagemaker/lineage/test_endpoint_context.py -vv
collected 6 items tests/integ/sagemaker/lineage/test_endpoint_context.py::test_pipeline_execution_arn PASSED [100%]

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

  • I have read the CONTRIBUTING doc
  • I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
  • I used the commit message format described in CONTRIBUTING
  • I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
  • I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
  • I have checked that my tests are not configured for a specific region or account (if appropriate)
  • I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@yzhu0 yzhu0 requested review from navinsoni, mufaddal-rohawala and a team January 6, 2022 17:37
@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 95fae83
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@yzhu0 yzhu0 force-pushed the Lineage_endpoints_modelPackageGroup branch from 95fae83 to 9813603 Compare January 6, 2022 17:59
@yzhu0 yzhu0 requested review from danabens and staubhp January 6, 2022 18:00
@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 9813603
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: 95fae83
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: 95fae83
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: 9813603
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: 9813603
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: 9813603
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@yzhu0 yzhu0 force-pushed the Lineage_endpoints_modelPackageGroup branch from 9813603 to c355230 Compare January 7, 2022 05:53
@yzhu0 yzhu0 changed the base branch from master to dev January 7, 2022 05:54
@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: c355230
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: c355230
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: c355230
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: c355230
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@yzhu0 yzhu0 force-pushed the Lineage_endpoints_modelPackageGroup branch from c355230 to 605e239 Compare January 7, 2022 07:30
@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 605e239
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: 605e239
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: 605e239
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: 605e239
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@yzhu0 yzhu0 requested a review from staubhp January 7, 2022 18:15
@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 5cc810c
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 5cc810c
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: 5cc810c
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@@ -382,6 +382,57 @@ def training_job_arns(
training_job_arns.append(trial_component["Source"]["SourceArn"])
return training_job_arns

def processing_job_arns(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add unit tests for this.

processing_job_arns.append(trial_component["Source"]["SourceArn"])
return processing_job_arns

def trial_components_arns(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add unit tests for this.

Returns:
str: A pipeline execution ARN.
"""
# return self.properties["PipelineExecutionArn"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this needed? Please remove commented out code prior to merge.

@@ -159,11 +160,11 @@ def __init__(
def _to_request_dict(self):
"""Convert the lineage filter to its API representation."""
filter_request = {}
if self.entities:
if self.sources:
filter_request["Types"] = list(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this a bug before?

Copy link
Contributor

@shreyapandit shreyapandit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs additional tests and other minor feedback to be addressed

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 1031517
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: 1031517
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: 1031517
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@yzhu0 yzhu0 force-pushed the Lineage_endpoints_modelPackageGroup branch from 1031517 to 0868591 Compare January 18, 2022 23:05
@codecov-commenter
Copy link

codecov-commenter commented Jan 18, 2022

Codecov Report

Merging #2830 (0868591) into dev (b796a00) will increase coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##              dev    #2830   +/-   ##
=======================================
  Coverage   88.96%   88.96%           
=======================================
  Files         178      178           
  Lines       15749    15752    +3     
=======================================
+ Hits        14011    14014    +3     
  Misses       1738     1738           
Impacted Files Coverage Δ
src/sagemaker/lineage/context.py 81.98% <100.00%> (+0.50%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b796a00...0868591. Read the comment docs.

@yzhu0 yzhu0 requested review from mufaddal-rohawala and shreyapandit and removed request for danabens January 18, 2022 23:11
@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 0868591
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: 0868591
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: 0868591
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Copy link
Contributor

@jeniyat jeniyat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me

@yzhu0
Copy link
Contributor Author

yzhu0 commented Jan 19, 2022

Needs additional tests and other minor feedback to be addressed

addressed all

@shreyapandit shreyapandit merged commit 9d25fcf into aws:dev Jan 19, 2022
shreyapandit added a commit that referenced this pull request Feb 3, 2022
* feature: allow conditional parellel builds (#2727)

* fix endpoint bug (#2772)

Co-authored-by: Basil Beirouti <[email protected]>

* fix: local mode - support relative file structure (#2768)

* prepare release v2.72.0

* update development version to v2.72.1.dev0

* fix: Set ProcessingStep upload locations deterministically to avoid c… (#2790)

* fix: Prevent repack_model script from referencing nonexistent directories (#2755)

Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: Ahsan Khan <[email protected]>

* fix: S3Input - add support for instance attributes (#2754)

* fix: typos and broken link (#2765)

Co-authored-by: Shreya Pandit <[email protected]>

* prepare release v2.72.1

* update development version to v2.72.2.dev0

* fix: Model Registration with BYO scripts (#2797)

Co-authored-by: Basil Beirouti <[email protected]>
Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: Ahsan Khan <[email protected]>
Co-authored-by: Mufaddal Rohawala <[email protected]>
Co-authored-by: Basil Beirouti <[email protected]>
Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: Shreya Pandit <[email protected]>

* fix: Add ContentType in test_auto_ml_describe

* fix: Re-deploy static integ test endpoint if it is not found

* documentation :SageMaker model parallel library 1.6.0 API doc (#2814)

* update smdmp change log, archive api doc for 1.4.0 and 1.5.0

* add no-index flags

* finish api doc archive

* fix: Set ProcessingStep upload locations deterministically to avoid c… (#2790)

* fix: Prevent repack_model script from referencing nonexistent directories (#2755)

Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: Ahsan Khan <[email protected]>

* fix: S3Input - add support for instance attributes (#2754)

* fix: typos and broken link (#2765)

Co-authored-by: Shreya Pandit <[email protected]>

* add all api docs

* add appendix, fix links

* structural changes, fix links

* incorporate feedback

* prepare release v2.72.1

* update development version to v2.72.2.dev0

Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: Ahsan Khan <[email protected]>
Co-authored-by: Mufaddal Rohawala <[email protected]>
Co-authored-by: Mohamed Ali Jamaoui <[email protected]>
Co-authored-by: Shreya Pandit <[email protected]>
Co-authored-by: ci <ci>
Co-authored-by: Jeniya Tabassum <[email protected]>

* fix: fix kmeans test deletion sequence, increment lineage statics (#2815)

* fix: Increment static lineage pipeline (#2817)

* fix: Update CHANGELOG.md (#2832)

* prepare release v2.72.2

* update development version to v2.72.3.dev0

* change: update master from dev (#2836)

Co-authored-by: Basil Beirouti <[email protected]>
Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: Ahsan Khan <[email protected]>
Co-authored-by: Mufaddal Rohawala <[email protected]>
Co-authored-by: Basil Beirouti <[email protected]>
Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: Shreya Pandit <[email protected]>
Co-authored-by: Mohamed Ali Jamaoui <[email protected]>
Co-authored-by: ci <ci>
Co-authored-by: Jeniya Tabassum <[email protected]>
Co-authored-by: sreedes <[email protected]>
Co-authored-by: Navin Soni <[email protected]>
Co-authored-by: Miyoung <[email protected]>
Co-authored-by: Ameen Khan <[email protected]>
Co-authored-by: Zhankui Lu <[email protected]>
Co-authored-by: Xiaoguang Chen <[email protected]>
Co-authored-by: Jonathan Guinegagne <[email protected]>
Co-authored-by: Zhankui Lu <[email protected]>
Co-authored-by: Yifei Zhu <[email protected]>
Co-authored-by: Qingzi-Lan <[email protected]>

* prepare release v2.72.3

* update development version to v2.72.4.dev0

* fix: fixes unnecessary session call while generating pipeline definition for lambda step (#2824)

* feature: Add models_v2 under lineage context (#2800)

* feature: enable python 3.9 (#2802)

Co-authored-by: Ahsan Khan <[email protected]>

* change: Update CHANGELOG.md (#2842)

* fix: update pricing link (#2805)

Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: Ahsan Khan <[email protected]>
Co-authored-by: Shreya Pandit <[email protected]>
Co-authored-by: Basil Beirouti <[email protected]>
Co-authored-by: Mufaddal Rohawala <[email protected]>
Co-authored-by: Basil Beirouti <[email protected]>
Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: Mohamed Ali Jamaoui <[email protected]>
Co-authored-by: ci <ci>
Co-authored-by: Jeniya Tabassum <[email protected]>
Co-authored-by: sreedes <[email protected]>
Co-authored-by: Navin Soni <[email protected]>
Co-authored-by: Miyoung <[email protected]>
Co-authored-by: Ameen Khan <[email protected]>
Co-authored-by: Zhankui Lu <[email protected]>
Co-authored-by: Navin Soni <[email protected]>
Co-authored-by: Xiaoguang Chen <[email protected]>
Co-authored-by: Jonathan Guinegagne <[email protected]>
Co-authored-by: Zhankui Lu <[email protected]>
Co-authored-by: Yifei Zhu <[email protected]>
Co-authored-by: Qingzi-Lan <[email protected]>

* doc: Document the available ExecutionVariables (#2807)

* fix: Remove duplicate vertex/edge in query lineage (#2784)

* feature: Support model pipelines in CreateModelStep (#2845)

Co-authored-by: Payton Staub <[email protected]>

* feature: support JsonGet/Join parameterization in tuning step Hyperparameters (#2833)

* doc: Enhance smddp 1.2.2 doc (#2852)

* feature: support checkpoint to be passed from estimator (#2849)

Co-authored-by: marckarp <[email protected]>

* fix: allow kms_key to be passed for processing step (#2779)

* feature: Adds support for Serverless inference (#2831)

* feature: Add support for SageMaker lineage queries in action (#2853)

* feature: Adds Lineage queries in artifact, context and trial components (#2838)

* feature: Add EMRStep support in Sagemaker pipeline (#2848)

Co-authored-by: chenxy <[email protected]>

* prepare release v2.73.0

* update development version to v2.73.1.dev0

* feature: Add support for SageMaker lineage queries context (#2830)

* fix: support specifying a facet by its column index

Currently the Clarify BiasConfig only accepts facet name. Actually
Clarify analysis configuration supports both name and index. This
commit adds the same support to BiasConfig.

* doc: more documentation for serverless inference (#2859)

* prepare release v2.74.0

* update development version to v2.74.1.dev0

* Add deprecation warning in Clarify DataConfig (#2847)

* feature: Update instance types for integ test (#2881)

* feature: Adds support for async inference (#2846)

* fix: update to incorporate black v22, pin tox versions (#2889)

Co-authored-by: Mufaddal Rohawala <[email protected]>

* make black happy

Co-authored-by: Mufaddal Rohawala <[email protected]>
Co-authored-by: Basil Beirouti <[email protected]>
Co-authored-by: Basil Beirouti <[email protected]>
Co-authored-by: ci <ci>
Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: Payton Staub <[email protected]>
Co-authored-by: Ahsan Khan <[email protected]>
Co-authored-by: Mohamed Ali Jamaoui <[email protected]>
Co-authored-by: Shreya Pandit <[email protected]>
Co-authored-by: sreedes <[email protected]>
Co-authored-by: Navin Soni <[email protected]>
Co-authored-by: Miyoung <[email protected]>
Co-authored-by: Jeniya Tabassum <[email protected]>
Co-authored-by: Ameen Khan <[email protected]>
Co-authored-by: Zhankui Lu <[email protected]>
Co-authored-by: Xiaoguang Chen <[email protected]>
Co-authored-by: Jonathan Guinegagne <[email protected]>
Co-authored-by: Zhankui Lu <[email protected]>
Co-authored-by: Yifei Zhu <[email protected]>
Co-authored-by: Qingzi-Lan <[email protected]>
Co-authored-by: Xinghan Chen <[email protected]>
Co-authored-by: Navin Soni <[email protected]>
Co-authored-by: Tulio Casagrande <[email protected]>
Co-authored-by: jerrypeng7773 <[email protected]>
Co-authored-by: marckarp <[email protected]>
Co-authored-by: marckarp <[email protected]>
Co-authored-by: jayatalr <[email protected]>
Co-authored-by: bhaoz <[email protected]>
Co-authored-by: Ethan Cheng <[email protected]>
Co-authored-by: chenxy <[email protected]>
Co-authored-by: Xiaoguang Chen <[email protected]>
Co-authored-by: keerthanvasist <[email protected]>
Co-authored-by: Mufaddal Rohawala <[email protected]>
Co-authored-by: Shreya Pandit <[email protected]>
@yzhu0 yzhu0 deleted the Lineage_endpoints_modelPackageGroup branch February 21, 2022 21:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants