fix: Add SELinux label to local docker volumes #3790

sodre · 2023-04-14T18:56:36Z

Issue #, if available:

Description of changes:
Add the SELinux ":z" option to the docker volume mounts. Without this setting the mounted volume is not visible inside the container. More documentation about the SELabel mount options and docker can be found in this blog-post

Testing done:
Ran sagemaker local in an SELinux enabled workstation.

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sodre · 2023-05-04T22:05:18Z

@knikure, this PR is ready for review

sodre

Added notes for reviewer.

sodre · 2023-05-04T22:09:39Z

src/sagemaker/local/image.py

@@ -49,6 +49,9 @@
 TRAINING_JOB_NAME_ENV_NAME = "TRAINING_JOB_NAME"
 S3_ENDPOINT_URL_ENV_NAME = "S3_ENDPOINT_URL"

+# SELinux Enabled
+SELINUX_ENABLED = os.environ.get("SAGEMAKER_LOCAL_SELINUX_ENABLED", "False").lower() in ["1", "true", "yes"]


I used an environment variable because I could not find a pythonic way to check if the system is running SELinux. The selinux package in pypi failed on my workstation when installed inside a conda environment, so I am discarding that option.

sodre · 2023-05-04T22:38:52Z

tests/unit/sagemaker/local/test_local_image.py

@@ -513,6 +513,7 @@ def test_train_local_code(get_data_source_instance, tmpdir, sagemaker_session):
            assert config["services"][h]["image"] == image
            assert config["services"][h]["command"] == "train"
            volumes = config["services"][h]["volumes"]
+            volumes = [v[:-2] if v.endswith(":z") else v for v in volumes]


Remove the :z SELinux label while reading the volume back from compose_data

sodre · 2023-05-04T22:39:37Z

src/sagemaker/local/image.py

@@ -349,6 +352,7 @@ def retrieve_artifacts(self, compose_data, output_data_config, job_name):
        # Gather the artifacts from all nodes into artifacts/model and artifacts/output
        for host in self.hosts:
            volumes = compose_data["services"][str(host)]["volumes"]
+            volumes = [v[:-2] if v.endswith(":z") else v for v in volumes]


Remove the :z SELinux label while reading the volume back from compose_data

knikure

/bot run all

sagemaker-bot · 2023-05-05T16:15:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: cc75bfb
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

knikure · 2023-05-05T16:55:35Z

@knikure, this PR is ready for review

@sodre Can you fix pylint and other errors listed in unit-tests build logs?

sagemaker-bot · 2023-05-05T17:02:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: cc75bfb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-05-05T17:07:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: cc75bfb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-05-05T17:07:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: cc75bfb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-05-05T17:19:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: cc75bfb
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sodre · 2023-05-12T14:06:06Z

/bot run all

sodre · 2023-05-12T14:07:48Z

@knikure, I fixed the lint issues. I don't think the failed integration tests are related to the change I made. Can you trigger the CI/CD again?

This flag can be used to turn on/off the SELinux labels for docker volumes

knikure

/bot run pr, slow-tests

sagemaker-bot · 2023-05-16T21:48:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 41e6187
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-05-16T22:07:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 41e6187
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-05-16T22:10:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 41e6187
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2023-05-16T22:18:06Z

Codecov Report

Merging #3790 (41e6187) into master (ca2d1c1) will decrease coverage by 0.71%.
The diff coverage is 98.59%.

@@            Coverage Diff             @@
##           master    #3790      +/-   ##
==========================================
- Coverage   89.95%   89.25%   -0.71%     
==========================================
  Files        1140      267     -873     
  Lines      105316    26061   -79255     
==========================================
- Hits        94741    23261   -71480     
+ Misses      10575     2800    -7775

Impacted Files	Coverage Δ
src/sagemaker/inputs.py	`100.00% <ø> (ø)`
src/sagemaker/remote_function/client.py	`96.90% <ø> (ø)`
...rc/sagemaker/remote_function/core/serialization.py	`96.07% <97.67%> (ø)`
src/sagemaker/experiments/run.py	`99.14% <100.00%> (ø)`
...ference_recommender/inference_recommender_mixin.py	`95.00% <100.00%> (ø)`
src/sagemaker/local/image.py	`76.48% <100.00%> (ø)`
.../sagemaker/remote_function/core/stored_function.py	`100.00% <100.00%> (ø)`
src/sagemaker/remote_function/errors.py	`100.00% <100.00%> (ø)`
src/sagemaker/remote_function/invoke_function.py	`77.35% <100.00%> (ø)`
src/sagemaker/remote_function/job.py	`88.93% <100.00%> (ø)`

... and 1397 files with indirect coverage changes

sagemaker-bot · 2023-05-17T00:00:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 41e6187
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-05-17T05:00:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 41e6187
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-05-17T18:41:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 41e6187
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-05-17T20:56:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 41e6187
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sodre requested a review from a team as a code owner April 14, 2023 18:56

sodre requested review from knikure and removed request for a team April 14, 2023 18:56

sodre marked this pull request as draft April 14, 2023 18:57

knikure self-assigned this Apr 24, 2023

knikure force-pushed the f/sm-local-selinux branch from 16d174e to 047e158 Compare April 24, 2023 20:20

sodre force-pushed the f/sm-local-selinux branch from 734b7bf to 4bae16e Compare May 4, 2023 22:02

sodre marked this pull request as ready for review May 4, 2023 22:02

sodre commented May 4, 2023

View reviewed changes

knikure reviewed May 5, 2023

View reviewed changes

sodre force-pushed the f/sm-local-selinux branch from 2453ae9 to ba4fc0f Compare May 12, 2023 13:54

sodre added 7 commits May 16, 2023 13:58

fix: 🐛 Add SELinux label to local docker volumes

8e07787

fix: 🔒 Use a shared Docker volume in sagemaker.local

fb830f4

feat: 🚸 Add SAGEMAKER_LOCAL_SELINUX_ENABLED environment variable

550f525

This flag can be used to turn on/off the SELinux labels for docker volumes

test: ✅ Add unit tests for SELinux usecase

10cae85

test: 💚 Remove :z from volumes in unit test assertion

98ef1ef

fix: 🐛 Remove SELinux label when parsing volumes from compose_data

253cce4

chore: 🚨 Fix lint errors

41e6187

knikure force-pushed the f/sm-local-selinux branch from a129ece to 41e6187 Compare May 16, 2023 20:58

knikure reviewed May 16, 2023

View reviewed changes

knikure approved these changes May 17, 2023

View reviewed changes

knikure merged commit 0809b05 into aws:master May 17, 2023

sodre deleted the f/sm-local-selinux branch May 24, 2023 18:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Add SELinux label to local docker volumes #3790

fix: Add SELinux label to local docker volumes #3790

sodre commented Apr 14, 2023 •

edited

Loading

sodre commented May 4, 2023

sodre left a comment

sodre May 4, 2023

sodre May 4, 2023

sodre May 4, 2023

knikure left a comment

sagemaker-bot commented May 5, 2023

knikure commented May 5, 2023

sagemaker-bot commented May 5, 2023

sagemaker-bot commented May 5, 2023

sagemaker-bot commented May 5, 2023

sagemaker-bot commented May 5, 2023

sodre commented May 12, 2023

sodre commented May 12, 2023

knikure left a comment •

edited

Loading

sagemaker-bot commented May 16, 2023

sagemaker-bot commented May 16, 2023

sagemaker-bot commented May 16, 2023

codecov-commenter commented May 16, 2023 •

edited

Loading

sagemaker-bot commented May 17, 2023

sagemaker-bot commented May 17, 2023

sagemaker-bot commented May 17, 2023

sagemaker-bot commented May 17, 2023

fix: Add SELinux label to local docker volumes #3790

fix: Add SELinux label to local docker volumes #3790

Conversation

sodre commented Apr 14, 2023 • edited Loading

Merge Checklist

General

Tests

sodre commented May 4, 2023

sodre left a comment

Choose a reason for hiding this comment

sodre May 4, 2023

Choose a reason for hiding this comment

sodre May 4, 2023

Choose a reason for hiding this comment

sodre May 4, 2023

Choose a reason for hiding this comment

knikure left a comment

Choose a reason for hiding this comment

sagemaker-bot commented May 5, 2023

AWS CodeBuild CI Report

knikure commented May 5, 2023

sagemaker-bot commented May 5, 2023

AWS CodeBuild CI Report

sagemaker-bot commented May 5, 2023

AWS CodeBuild CI Report

sagemaker-bot commented May 5, 2023

AWS CodeBuild CI Report

sagemaker-bot commented May 5, 2023

AWS CodeBuild CI Report

sodre commented May 12, 2023

sodre commented May 12, 2023

knikure left a comment • edited Loading

Choose a reason for hiding this comment

sagemaker-bot commented May 16, 2023

AWS CodeBuild CI Report

sagemaker-bot commented May 16, 2023

AWS CodeBuild CI Report

sagemaker-bot commented May 16, 2023

AWS CodeBuild CI Report

codecov-commenter commented May 16, 2023 • edited Loading

Codecov Report

sagemaker-bot commented May 17, 2023

AWS CodeBuild CI Report

sagemaker-bot commented May 17, 2023

AWS CodeBuild CI Report

sagemaker-bot commented May 17, 2023

AWS CodeBuild CI Report

sagemaker-bot commented May 17, 2023

AWS CodeBuild CI Report

sodre commented Apr 14, 2023 •

edited

Loading

knikure left a comment •

edited

Loading

codecov-commenter commented May 16, 2023 •

edited

Loading