Skip to content

feature: add support for lineage visualization using pyvis #3825

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 82 commits into from
May 4, 2023
Merged

feature: add support for lineage visualization using pyvis #3825

merged 82 commits into from
May 4, 2023

Conversation

jkasiraj
Copy link
Contributor

@jkasiraj jkasiraj commented Apr 28, 2023 •
edited
Loading

Issue #, if available:

Description of changes:
Add visualize function to LineageQueryResult to visualize and interact with lineage graph

Previous prs to feature branch:

Example usage:

lineage = sagemaker.lineage.query.LineageQuery(sagemaker_session)
query_results = lineage.query(start_arns=[graph_startarn])
query_results.visualize(pth='pyvisSample.html')
  • Hover on Node

image

  • Example graph

image

Testing done:
tox -e black-format & tox -e flake8,pylint,twine,black-check,docstyle --parallel all
tox -r -e py39 tests/integ/sagemaker/lineage/test_lineage_visualize.py
tox -e py39 tests/unit/sagemaker/lineage/test_query.py

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

  • I have read the CONTRIBUTING doc
  • I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
  • I used the commit message format described in CONTRIBUTING
  • I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
  • I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
  • I have checked that my tests are not configured for a specific region or account (if appropriate)
  • I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Yi-Ting Lee and others added 30 commits July 1, 2022 12:59
feature: add __str__ methods to queryLineage output classes
77965db
@ytlee93
Merge pull request #1 from ytlee93/master
6d005e2 
 feature: add __str__ methods to queryLineage output classes
@Snoyark
Merge branch 'aws:master' into master
e0c59c2
feature: query lineage visualizer for general case
7359b3d 
edge.association_type added

style changes of graph
startarn added to lineageQueryResult
d524416
color node by lineage entity
acf64f4
identify startarn node by shape
e07e83c
Add code comments
687d6b6
Double sided arrows handled
e3a4c9d
legend added
1808f87
feature: query lineage visualizer for general case
79b193a 
edge.association_type added

style changes of graph

DashVisualizer class added

Try except when importing visual modules
@ytlee93
Merge pull request #3 from ytlee93/master
cbe445e 
feature: query lineage visualizer for general case
try except raise
60904d5
startarn added
9125eb9
add get element function
3db7fa1
add DashVisualizer class
c135ab9
style check
daa5cc4
Merge branch 'master' into advanced-style
0adc4db
feature: query lineage visualizer advanced styling & interactive comp…
28a9eeb 
…onent handle
@ytlee93
Merge pull request #1 from Snoyark/master
67346a7 
Merge pull request #3 from ytlee93/master
add functions that generate html components and style selectors
7c2b0c3
inject graph data to DashVisualizer task
41d2453
@ytlee93
Merge pull request #4 from ytlee93/master
e6078a9 
feature: query lineage visualizer advanced styling & interactive component handling
test_lineage_visualize.py created
b74e861
@ytlee93
Merge pull request #2 from Snoyark/master
0b35553 
Merge pull request #4 from ytlee93/master
PyvisVisualizer added
b6b5ee4
modify options
794fb57
change: change visualization to using pyvis library
baeaace
Merge branch 'master' into large-graph-test
c3d2ff7
pyvis import issue on running lineage test
578acdf
@jkasiraj jkasiraj force-pushed the lineageVisualize branch from c5125c0 to 8be84f1 Compare May 1, 2023 20:37
@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 8be84f1
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jkasiraj jkasiraj force-pushed the lineageVisualize branch from 8be84f1 to 32e7383 Compare May 1, 2023 20:53
@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: 32e7383
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 32e7383
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 32e7383
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: 32e7383
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: 383cae0
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: 383cae0
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: 383cae0
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 383cae0
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 383cae0
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jkasiraj
Copy link
Contributor Author

jkasiraj commented May 1, 2023

sagemaker-python-sdk-slow-tests failures are unrelated to pr:

987 | FAILED tests/integ/test_inference_pipeline.py::test_inference_pipeline_model_deploy_and_update_endpoint
988 | ERROR tests/integ/test_model_quality_monitor.py::test_run_model_quality_monitor
989 | ERROR tests/integ/test_model_quality_monitor.py::test_run_model_quality_monitor_baseline

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 6460e9e
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: 6460e9e
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: 6460e9e
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 6460e9e
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 74f90a3
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: 74f90a3
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: 74f90a3
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 74f90a3
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: 74f90a3
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@trajanikant trajanikant merged commit 798cc76 into aws:master May 4, 2023
@jkasiraj jkasiraj mentioned this pull request May 4, 2023
Closed
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@danabens danabens danabens approved these changes

@trajanikant trajanikant Awaiting requested review from trajanikant trajanikant is a code owner automatically assigned from aws/sagemaker-ml-frameworks

Assignees

@danabens danabens

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@jkasiraj @sagemaker-bot @codecov-commenter @danabens @trajanikant @ytlee93 @Snoyark