From 3b5e52f4bc29a0da60a0b19290ef7910ff1fc492 Mon Sep 17 00:00:00 2001 From: default Date: Thu, 31 Oct 2024 09:40:43 +0000 Subject: [PATCH 1/9] feat(docs): document build secrets --- docs/build-secrets.md | 82 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 docs/build-secrets.md diff --git a/docs/build-secrets.md b/docs/build-secrets.md new file mode 100644 index 00000000..ce8a4e95 --- /dev/null +++ b/docs/build-secrets.md @@ -0,0 +1,82 @@ +# Build Secrets + +Envbuilder supports [build secrets](https://docs.docker.com/reference/dockerfile/#run---mounttypesecret). For envbuilder, build secrets are useful when you need to use sensitive information during the image build process and: +* the secrets should not be present in the built image. +* the secrets should not be accessible in the container after its built has concluded. + +If your Dockerfile contains directives of the form `RUN --mount=type=secret,...`, Envbuilder will attempt to mount build secrets as specified. Envbuilder does need to find these secrets. Unlike the `docker build` command, Envbuilder does not support the `--secret` flag. Instead, Envbuilder collects build secrets from environment variables prefixed with `ENVBUILDER_BUILD_SECRET_`. +These build secrets will not be present in any cached layers or images that are pushed to an image repository. Nor will they be available at run time. + +## Example + +To illustrate build secrets in envbuilder, let's build, push and run a container locally. These concepts will transfer to Kubernetes or other containerised environments. Note that this example is for illustrative purposes only and is not fit for production use. Production considerations are discussed in the next section. + +First, start a local docker registry, so that we can push and inspect the built image: +```bash +docker run -d -p 5000:5000 --name envbuilder-registry --volume $(PWD)/.registry-cache:/var/lib/registry registry:2; +``` + +Then, build an image based on this Dockerfile: + +```Dockerfile +FROM alpine:latest + +RUN --mount=type=secret,id=FOO,env cat $FOO > /foo_secret.txt +RUN --mount=type=secret,id=BAR,dst=/tmp/bar.secret cat /tmp/bar.secret > /bar_secret.txt +``` +using this command: +```bash +docker run -it --rm \ + # Set build secrets: + -e ENVBUILDER_SECRET_FOO='foo' \ + -e ENVBUILDER_SECRET_BAR='bar' \ + # Cache image layers in a local registry + -e CACHE_REPO='localhost:5000'\ + # Push the image that envbuilder builds + -e PUSH_IMAGE='1' \ + # Configure the envbuilder workspace + -e ENVBUILDER_INIT_SCRIPT='/bin/sh' \ + -e ENVBUILDER_WORKSPACE_FOLDER=/workspace \ + -v $PWD:/workspace \ + -v /var/run/docker.sock:/var/run/docker.sock \ + ghcr.io/coder/envbuilder:latest +``` + +This will result in a shell session inside the built container. +You can now verify two things: +* The secrets provided to build are not available once the container is running. +* The secrets were still useful during the build: +```bash +cat /foo_secret.txt +cat /bar_secret.txt +``` + +Once done, exit the container and proceed to the next step. + +We have verified that build secrets do not persist into the runtime container environment. Let's now verify that they also do not persist into the built image. To do this, pull the image and save it as a .tar file. Then, extract it +and inspect its manifests and layers. + +```bash +# Determine the image name and tag +curl ... +# Pull and save to disk +docker pull localhost:5000/ +docker save -o image.tar localhost:5000/ +# Inspect the contents + +``` + +## Security and Production Use +The example above ignores various security concerns for the sake of simple illustration. To use build secrets securely, consider these factors: + +### Who should be able to access build secrets, when and where? +The secure way to use build secrets with envbuilder is to deny users access to the platform that hosts envbuilder. Only grant access to the envbuilder container once it has concluded its build, using a trusted non-platform channel like ssh or the coder agent running inside the container. Once control has been handed to such a runtime container process, envbuilder will have cleared all secrets that it set from the container. + +Anyone with sufficient access to attach directly to the container (eg. using `kubectl`), will be able to read build secrets if they attach to the container before it has concluded its build. Anyone with sufficient access to the platform that hosts the envbuilder container will also be able to read these build secrets from where the platform stores them. This is true for other build systems, and software in general. + +If secrets should be accessible at runtime, do not use build secrets. Rather, mount the secret data using a volume or environment variable. Envbuilder will not include mounted volumes in the image that it pushes to any cache repositories, but they will still be available to users that connect to the container. + +### Container Management beyond Envbuilder's control +Container orchestration systems mount certain artifacts into containers for various reasons. It is possible that some of these might grant indirect access to build secrets. Consider kubernetes. It will mount a service account token into running containers. Depending on the access granted to this service account token, it may be possible to read build secrets and other sensitive data using the kubernetes API. This should not be possible by default, but envbuilder cannot provide such a guarantee. + +When building a system that uses envbuilder, ensure that your platform does not expose unintended secret information to the container. \ No newline at end of file From 2ebca758862ded465e375ddbada5d4f44ec3a25c Mon Sep 17 00:00:00 2001 From: Sas Swart Date: Thu, 31 Oct 2024 13:32:02 +0000 Subject: [PATCH 2/9] feat(envbuilder.go): document build secrets --- docs/build-secrets.md | 44 ++++++++++++++----------------------------- envbuilder.go | 6 ++++++ 2 files changed, 20 insertions(+), 30 deletions(-) diff --git a/docs/build-secrets.md b/docs/build-secrets.md index ce8a4e95..4f1b5366 100644 --- a/docs/build-secrets.md +++ b/docs/build-secrets.md @@ -13,7 +13,7 @@ To illustrate build secrets in envbuilder, let's build, push and run a container First, start a local docker registry, so that we can push and inspect the built image: ```bash -docker run -d -p 5000:5000 --name envbuilder-registry --volume $(PWD)/.registry-cache:/var/lib/registry registry:2; +docker run --rm -d -p 5000:5000 --name envbuilder-registry registry:2 ``` Then, build an image based on this Dockerfile: @@ -21,54 +21,38 @@ Then, build an image based on this Dockerfile: ```Dockerfile FROM alpine:latest -RUN --mount=type=secret,id=FOO,env cat $FOO > /foo_secret.txt -RUN --mount=type=secret,id=BAR,dst=/tmp/bar.secret cat /tmp/bar.secret > /bar_secret.txt +RUN --mount=type=secret,id=FOO,env cat $FOO > /foo_secret_hash.txt +RUN --mount=type=secret,id=BAR,dst=/tmp/bar.secret cat /tmp/bar.secret > /bar_secret_hash.txt ``` using this command: ```bash docker run -it --rm \ - # Set build secrets: - -e ENVBUILDER_SECRET_FOO='foo' \ - -e ENVBUILDER_SECRET_BAR='bar' \ - # Cache image layers in a local registry - -e CACHE_REPO='localhost:5000'\ - # Push the image that envbuilder builds - -e PUSH_IMAGE='1' \ - # Configure the envbuilder workspace + -e ENVBUILDER_BUILD_SECRET_FOO='envbuilder-test-secret-foo' \ + -e ENVBUILDER_BUILD_SECRET_BAR='envbuilder-test-secret-bar' \ -e ENVBUILDER_INIT_SCRIPT='/bin/sh' \ - -e ENVBUILDER_WORKSPACE_FOLDER=/workspace \ - -v $PWD:/workspace \ - -v /var/run/docker.sock:/var/run/docker.sock \ + -e ENVBUILDER_CACHE_REPO=$(docker inspect envbuilder-registry | jq -r '.[].NetworkSettings.IPAddress'):5000/test-container \ + -e ENVBUILDER_PUSH_IMAGE=0 \ + -v $PWD:/workspaces/empty \ ghcr.io/coder/envbuilder:latest ``` This will result in a shell session inside the built container. You can now verify two things: -* The secrets provided to build are not available once the container is running. +* The secrets provided to build are not available once the container is running. They are no longer on disk, nor are they in the process environment, or in `/proc/self/environ`. * The secrets were still useful during the build: ```bash cat /foo_secret.txt cat /bar_secret.txt ``` -Once done, exit the container and proceed to the next step. - -We have verified that build secrets do not persist into the runtime container environment. Let's now verify that they also do not persist into the built image. To do this, pull the image and save it as a .tar file. Then, extract it -and inspect its manifests and layers. - -```bash -# Determine the image name and tag -curl ... -# Pull and save to disk -docker pull localhost:5000/ -docker save -o image.tar localhost:5000/ -# Inspect the contents - -``` - ## Security and Production Use The example above ignores various security concerns for the sake of simple illustration. To use build secrets securely, consider these factors: +### Build Secret Purpose and Management +Build secrets are meant for use cases where the secret should not be accessible from the built image, nor from the running container. If you need the secret at runtime, use a volume instead. Volumes that are mounted into a container will not be included in the final image, but still be available at runtime. + +Build secrets are only protected if they are not copied or moved from their location as designated in the `RUN` directive. If a build secret is used, care should be taken to ensure that it is not copied or otherwise persisted into an image layer beyond the control of Envbuilder. + ### Who should be able to access build secrets, when and where? The secure way to use build secrets with envbuilder is to deny users access to the platform that hosts envbuilder. Only grant access to the envbuilder container once it has concluded its build, using a trusted non-platform channel like ssh or the coder agent running inside the container. Once control has been handed to such a runtime container process, envbuilder will have cleared all secrets that it set from the container. diff --git a/envbuilder.go b/envbuilder.go index b621a85c..8bb9af69 100644 --- a/envbuilder.go +++ b/envbuilder.go @@ -1267,6 +1267,11 @@ func RunCacheProbe(ctx context.Context, opts options.Options) (v1.Image, error) if opts.CacheRepo != "" { destinations = append(destinations, opts.CacheRepo) } + + buildSecrets := options.GetBuildSecrets(os.Environ()) + // Ensure that build secrets do not make it into the runtime environment or the setup script: + options.ClearBuildSecretsFromProcessEnvironment() + kOpts := &config.KanikoOptions{ // Boilerplate! CustomPlatform: platforms.Format(platforms.Normalize(platforms.DefaultSpec())), @@ -1291,6 +1296,7 @@ func RunCacheProbe(ctx context.Context, opts options.Options) (v1.Image, error) }, ForceUnpack: true, BuildArgs: buildParams.BuildArgs, + BuildSecrets: buildSecrets, CacheRepo: opts.CacheRepo, Cache: opts.CacheRepo != "" || opts.BaseImageCacheDir != "", DockerfilePath: buildParams.DockerfilePath, From 7c42b45355a8c1ac1e366150b80385754b3f2aff Mon Sep 17 00:00:00 2001 From: Sas Swart Date: Thu, 31 Oct 2024 14:04:33 +0000 Subject: [PATCH 3/9] feat(envbuilder.go): document build secrets --- docs/build-secrets.md | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/docs/build-secrets.md b/docs/build-secrets.md index 4f1b5366..208d3d46 100644 --- a/docs/build-secrets.md +++ b/docs/build-secrets.md @@ -31,7 +31,7 @@ docker run -it --rm \ -e ENVBUILDER_BUILD_SECRET_BAR='envbuilder-test-secret-bar' \ -e ENVBUILDER_INIT_SCRIPT='/bin/sh' \ -e ENVBUILDER_CACHE_REPO=$(docker inspect envbuilder-registry | jq -r '.[].NetworkSettings.IPAddress'):5000/test-container \ - -e ENVBUILDER_PUSH_IMAGE=0 \ + -e ENVBUILDER_PUSH_IMAGE=1 \ -v $PWD:/workspaces/empty \ ghcr.io/coder/envbuilder:latest ``` @@ -41,8 +41,33 @@ You can now verify two things: * The secrets provided to build are not available once the container is running. They are no longer on disk, nor are they in the process environment, or in `/proc/self/environ`. * The secrets were still useful during the build: ```bash -cat /foo_secret.txt -cat /bar_secret.txt +cat /foo_secret_hash.txt +cat /bar_secret_hash.txt +``` + +### Verifying that images are secret free +To verify that the build image doesn't contain build secrets, run the following: + +```bash +docker pull localhost:5000/test-container:latest +docker save -o test-container.tar localhost:5000/test-container +mkdir -p test-container +tar -xf test-container.tar -C test-container/ +cd test-container +# Scan image layers for secrets: +find . -type f | xargs tar -xOf 2>/dev/null | strings | grep -rin "envbuilder-test-secret" +# Scan image manifests for secrets: +find . -type f | xargs -n1 grep -rinI 'envbuilder-test-secret' +cd ../ +``` + +The output of both find/grep commands should be empty. +To verify that it scans correctly, replace "envbuilder-test-secret" with "envbuilder" and rerun the commands. + +Having verified that no secrets were included in the image, we can now delete the artifacts that we saved to disk. +```bash +rm -r test-container +rm -r test-container.tar ``` ## Security and Production Use From 51cd4d4d6b2162ac8ec3d05344bd788c3314bb02 Mon Sep 17 00:00:00 2001 From: Sas Swart Date: Thu, 31 Oct 2024 17:37:21 +0000 Subject: [PATCH 4/9] feat(docs): document build secrets --- docs/build-secrets.md | 35 +++++++++++++++++------------------ 1 file changed, 17 insertions(+), 18 deletions(-) diff --git a/docs/build-secrets.md b/docs/build-secrets.md index 208d3d46..ab4c7feb 100644 --- a/docs/build-secrets.md +++ b/docs/build-secrets.md @@ -1,19 +1,18 @@ # Build Secrets -Envbuilder supports [build secrets](https://docs.docker.com/reference/dockerfile/#run---mounttypesecret). For envbuilder, build secrets are useful when you need to use sensitive information during the image build process and: +Envbuilder supports [build secrets](https://docs.docker.com/reference/dockerfile/#run---mounttypesecret). Build secrets are useful when you need to use sensitive information during the image build process and: * the secrets should not be present in the built image. -* the secrets should not be accessible in the container after its built has concluded. +* the secrets should not be accessible in the container after its build has concluded. -If your Dockerfile contains directives of the form `RUN --mount=type=secret,...`, Envbuilder will attempt to mount build secrets as specified. Envbuilder does need to find these secrets. Unlike the `docker build` command, Envbuilder does not support the `--secret` flag. Instead, Envbuilder collects build secrets from environment variables prefixed with `ENVBUILDER_BUILD_SECRET_`. -These build secrets will not be present in any cached layers or images that are pushed to an image repository. Nor will they be available at run time. +If your Dockerfile contains directives of the form `RUN --mount=type=secret,...`, Envbuilder will attempt to mount build secrets as specified in the directive. Unlike the `docker build` command, Envbuilder does not support the `--secret` flag. Instead, Envbuilder collects build secrets from environment variables prefixed with `Envbuilder_BUILD_SECRET_`. These build secrets will not be present in any cached layers or images that are pushed to an image repository. Nor will they be available at run time. ## Example -To illustrate build secrets in envbuilder, let's build, push and run a container locally. These concepts will transfer to Kubernetes or other containerised environments. Note that this example is for illustrative purposes only and is not fit for production use. Production considerations are discussed in the next section. +To illustrate build secrets in Envbuilder, let's build, push and run a container locally. These concepts will transfer to Kubernetes or other containerised environments. Note that this example is for illustrative purposes only and is not fit for production use. Production considerations are discussed in the next section. First, start a local docker registry, so that we can push and inspect the built image: ```bash -docker run --rm -d -p 5000:5000 --name envbuilder-registry registry:2 +docker run --rm -d -p 5000:5000 --name Envbuilder-registry registry:2 ``` Then, build an image based on this Dockerfile: @@ -27,19 +26,19 @@ RUN --mount=type=secret,id=BAR,dst=/tmp/bar.secret cat /tmp/bar.secret > /bar_se using this command: ```bash docker run -it --rm \ - -e ENVBUILDER_BUILD_SECRET_FOO='envbuilder-test-secret-foo' \ - -e ENVBUILDER_BUILD_SECRET_BAR='envbuilder-test-secret-bar' \ - -e ENVBUILDER_INIT_SCRIPT='/bin/sh' \ - -e ENVBUILDER_CACHE_REPO=$(docker inspect envbuilder-registry | jq -r '.[].NetworkSettings.IPAddress'):5000/test-container \ - -e ENVBUILDER_PUSH_IMAGE=1 \ + -e Envbuilder_BUILD_SECRET_FOO='envbuilder-test-secret-foo' \ + -e Envbuilder_BUILD_SECRET_BAR='envbuilder-test-secret-bar' \ + -e Envbuilder_INIT_SCRIPT='/bin/sh' \ + -e Envbuilder_CACHE_REPO=$(docker inspect Envbuilder-registry | jq -r '.[].NetworkSettings.IPAddress'):5000/test-container \ + -e Envbuilder_PUSH_IMAGE=1 \ -v $PWD:/workspaces/empty \ - ghcr.io/coder/envbuilder:latest + ghcr.io/coder/Envbuilder:latest ``` This will result in a shell session inside the built container. You can now verify two things: * The secrets provided to build are not available once the container is running. They are no longer on disk, nor are they in the process environment, or in `/proc/self/environ`. -* The secrets were still useful during the build: +* The secrets were still useful during the build. The following comnmands show that the secrets had side effects inside the build, without remaining in the image: ```bash cat /foo_secret_hash.txt cat /bar_secret_hash.txt @@ -62,7 +61,7 @@ cd ../ ``` The output of both find/grep commands should be empty. -To verify that it scans correctly, replace "envbuilder-test-secret" with "envbuilder" and rerun the commands. +To verify that it scans correctly, replace "envbuilder-test-secret" with "Envbuilder" and rerun the commands. It should find strings related to Envbuilder that are not secrets. Having verified that no secrets were included in the image, we can now delete the artifacts that we saved to disk. ```bash @@ -79,13 +78,13 @@ Build secrets are meant for use cases where the secret should not be accessible Build secrets are only protected if they are not copied or moved from their location as designated in the `RUN` directive. If a build secret is used, care should be taken to ensure that it is not copied or otherwise persisted into an image layer beyond the control of Envbuilder. ### Who should be able to access build secrets, when and where? -The secure way to use build secrets with envbuilder is to deny users access to the platform that hosts envbuilder. Only grant access to the envbuilder container once it has concluded its build, using a trusted non-platform channel like ssh or the coder agent running inside the container. Once control has been handed to such a runtime container process, envbuilder will have cleared all secrets that it set from the container. +The secure way to use build secrets with Envbuilder is to deny users access to the platform that hosts Envbuilder. Only grant access to the Envbuilder container once it has concluded its build, using a trusted non-platform channel like ssh or the coder agent running inside the container. Once control has been handed to such a runtime container process, Envbuilder will have cleared all secrets that it set from the container. -Anyone with sufficient access to attach directly to the container (eg. using `kubectl`), will be able to read build secrets if they attach to the container before it has concluded its build. Anyone with sufficient access to the platform that hosts the envbuilder container will also be able to read these build secrets from where the platform stores them. This is true for other build systems, and software in general. +Anyone with sufficient access to attach directly to the container (eg. using `kubectl`), will be able to read build secrets if they attach to the container before it has concluded its build. Anyone with sufficient access to the platform that hosts the Envbuilder container will also be able to read these build secrets from where the platform stores them. This is true for other build systems, and containerised software in general. If secrets should be accessible at runtime, do not use build secrets. Rather, mount the secret data using a volume or environment variable. Envbuilder will not include mounted volumes in the image that it pushes to any cache repositories, but they will still be available to users that connect to the container. ### Container Management beyond Envbuilder's control -Container orchestration systems mount certain artifacts into containers for various reasons. It is possible that some of these might grant indirect access to build secrets. Consider kubernetes. It will mount a service account token into running containers. Depending on the access granted to this service account token, it may be possible to read build secrets and other sensitive data using the kubernetes API. This should not be possible by default, but envbuilder cannot provide such a guarantee. +Container orchestration systems mount certain artifacts into containers for various reasons. It is possible that some of these might grant indirect access to build secrets. Consider kubernetes. It will mount a service account token into running containers. Depending on the access granted to this service account token, it may be possible to read build secrets and other sensitive data using the kubernetes API. This should not be possible by default, but Envbuilder cannot provide such a guarantee. -When building a system that uses envbuilder, ensure that your platform does not expose unintended secret information to the container. \ No newline at end of file +When building a system that uses Envbuilder, ensure that your platform does not expose unintended secret information to the container. \ No newline at end of file From dfb133c62ada0b01ac86e2254a4aa67ad3986f84 Mon Sep 17 00:00:00 2001 From: Sas Swart Date: Fri, 1 Nov 2024 13:48:09 +0000 Subject: [PATCH 5/9] fix(options): read build secrets the same way that we read other options --- docs/build-secrets.md | 11 ++- envbuilder.go | 12 +-- integration/integration_test.go | 32 +------- options/options.go | 9 +++ options/secrets.go | 46 ----------- options/secrets_test.go | 138 -------------------------------- options/testdata/options.golden | 4 + 7 files changed, 21 insertions(+), 231 deletions(-) delete mode 100644 options/secrets.go delete mode 100644 options/secrets_test.go diff --git a/docs/build-secrets.md b/docs/build-secrets.md index ab4c7feb..5f39c1d0 100644 --- a/docs/build-secrets.md +++ b/docs/build-secrets.md @@ -4,7 +4,7 @@ Envbuilder supports [build secrets](https://docs.docker.com/reference/dockerfile * the secrets should not be present in the built image. * the secrets should not be accessible in the container after its build has concluded. -If your Dockerfile contains directives of the form `RUN --mount=type=secret,...`, Envbuilder will attempt to mount build secrets as specified in the directive. Unlike the `docker build` command, Envbuilder does not support the `--secret` flag. Instead, Envbuilder collects build secrets from environment variables prefixed with `Envbuilder_BUILD_SECRET_`. These build secrets will not be present in any cached layers or images that are pushed to an image repository. Nor will they be available at run time. +If your Dockerfile contains directives of the form `RUN --mount=type=secret,...`, Envbuilder will attempt to mount build secrets as specified in the directive. Unlike the `docker build` command, Envbuilder does not support the `--secret` flag. Instead, Envbuilder collects build secrets from the `ENVBUILDER_BUILD_SECRETS` environment variable. These build secrets will not be present in any cached layers or images that are pushed to an image repository. Nor will they be available at run time. ## Example @@ -26,11 +26,10 @@ RUN --mount=type=secret,id=BAR,dst=/tmp/bar.secret cat /tmp/bar.secret > /bar_se using this command: ```bash docker run -it --rm \ - -e Envbuilder_BUILD_SECRET_FOO='envbuilder-test-secret-foo' \ - -e Envbuilder_BUILD_SECRET_BAR='envbuilder-test-secret-bar' \ - -e Envbuilder_INIT_SCRIPT='/bin/sh' \ - -e Envbuilder_CACHE_REPO=$(docker inspect Envbuilder-registry | jq -r '.[].NetworkSettings.IPAddress'):5000/test-container \ - -e Envbuilder_PUSH_IMAGE=1 \ + -e ENVBUILDER_BUILD_SECRETS='FOO=envbuilder-test-secret-foo,BAR=envbuilder-test-secret-bar' \ + -e ENVBUILDER_INIT_SCRIPT='/bin/sh' \ + -e ENVBUILDER_CACHE_REPO=$(docker inspect Envbuilder-registry | jq -r '.[].NetworkSettings.IPAddress'):5000/test-container \ + -e ENVBUILDER_PUSH_IMAGE=1 \ -v $PWD:/workspaces/empty \ ghcr.io/coder/Envbuilder:latest ``` diff --git a/envbuilder.go b/envbuilder.go index 8bb9af69..0603df46 100644 --- a/envbuilder.go +++ b/envbuilder.go @@ -531,10 +531,6 @@ func run(ctx context.Context, opts options.Options, execArgs *execArgsInfo) erro destinations = append(destinations, opts.CacheRepo) } - buildSecrets := options.GetBuildSecrets(os.Environ()) - // Ensure that build secrets do not make it into the runtime environment or the setup script: - options.ClearBuildSecretsFromProcessEnvironment() - kOpts := &config.KanikoOptions{ // Boilerplate! CustomPlatform: platforms.Format(platforms.Normalize(platforms.DefaultSpec())), @@ -559,7 +555,7 @@ func run(ctx context.Context, opts options.Options, execArgs *execArgsInfo) erro }, ForceUnpack: true, BuildArgs: buildParams.BuildArgs, - BuildSecrets: buildSecrets, + BuildSecrets: opts.BuildSecrets, CacheRepo: opts.CacheRepo, Cache: opts.CacheRepo != "" || opts.BaseImageCacheDir != "", DockerfilePath: buildParams.DockerfilePath, @@ -1268,10 +1264,6 @@ func RunCacheProbe(ctx context.Context, opts options.Options) (v1.Image, error) destinations = append(destinations, opts.CacheRepo) } - buildSecrets := options.GetBuildSecrets(os.Environ()) - // Ensure that build secrets do not make it into the runtime environment or the setup script: - options.ClearBuildSecretsFromProcessEnvironment() - kOpts := &config.KanikoOptions{ // Boilerplate! CustomPlatform: platforms.Format(platforms.Normalize(platforms.DefaultSpec())), @@ -1296,7 +1288,7 @@ func RunCacheProbe(ctx context.Context, opts options.Options) (v1.Image, error) }, ForceUnpack: true, BuildArgs: buildParams.BuildArgs, - BuildSecrets: buildSecrets, + BuildSecrets: opts.BuildSecrets, CacheRepo: opts.CacheRepo, Cache: opts.CacheRepo != "" || opts.BaseImageCacheDir != "", DockerfilePath: buildParams.DockerfilePath, diff --git a/integration/integration_test.go b/integration/integration_test.go index a576d3e9..08f81078 100644 --- a/integration/integration_test.go +++ b/integration/integration_test.go @@ -1113,36 +1113,6 @@ func TestUnsetOptionsEnv(t *testing.T) { } } -func TestUnsetSecretEnvs(t *testing.T) { - t.Parallel() - - // Ensures that a Git repository with a devcontainer.json is cloned and built. - srv := gittest.CreateGitServer(t, gittest.Options{ - Files: map[string]string{ - ".devcontainer/devcontainer.json": `{ - "name": "Test", - "build": { - "dockerfile": "Dockerfile" - }, - }`, - ".devcontainer/Dockerfile": "FROM " + testImageAlpine + "\nENV FROM_DOCKERFILE=foo", - }, - }) - ctr, err := runEnvbuilder(t, runOpts{env: []string{ - envbuilderEnv("GIT_URL", srv.URL), - envbuilderEnv("GIT_PASSWORD", "supersecret"), - options.EnvWithBuildSecretPrefix("FOO", "foo"), - envbuilderEnv("INIT_SCRIPT", "env > /root/env.txt && sleep infinity"), - }}) - require.NoError(t, err) - - output := execContainer(t, ctr, "cat /root/env.txt") - envsAvailableToInitScript := strings.Split(strings.TrimSpace(output), "\n") - - leftoverBuildSecrets := options.GetBuildSecrets(envsAvailableToInitScript) - require.Empty(t, leftoverBuildSecrets, "build secrets should not be available to init script") -} - func TestBuildSecrets(t *testing.T) { t.Parallel() @@ -1172,7 +1142,7 @@ func TestBuildSecrets(t *testing.T) { ctr, err := runEnvbuilder(t, runOpts{env: []string{ envbuilderEnv("GIT_URL", srv.URL), envbuilderEnv("GIT_PASSWORD", "supersecret"), - options.EnvWithBuildSecretPrefix("FOO", buildSecretVal), + envbuilderEnv("BUILD_SECRETS", fmt.Sprintf("FOO=%s", buildSecretVal)), }}) require.NoError(t, err) diff --git a/options/options.go b/options/options.go index efa61be7..01ed510d 100644 --- a/options/options.go +++ b/options/options.go @@ -88,6 +88,9 @@ type Options struct { // IgnorePaths is the comma separated list of paths to ignore when building // the workspace. IgnorePaths []string + // BuildSecrets is the list of secret environment variables to use when + // building the image. + BuildSecrets []string // SkipRebuild skips building if the MagicFile exists. This is used to skip // building when a container is restarting. e.g. docker stop -> docker start // This value can always be set to true - even if the container is being @@ -323,6 +326,12 @@ func (o *Options) CLI() serpent.OptionSet { Description: "The comma separated list of paths to ignore when " + "building the workspace.", }, + { + Flag: "build-secrets", + Env: WithEnvPrefix("BUILD_SECRETS"), + Value: serpent.StringArrayOf(&o.BuildSecrets), + Description: "The list of secret environment variables to use " + "when building the image.", + }, { Flag: "skip-rebuild", Env: WithEnvPrefix("SKIP_REBUILD"), diff --git a/options/secrets.go b/options/secrets.go deleted file mode 100644 index 9896fb36..00000000 --- a/options/secrets.go +++ /dev/null @@ -1,46 +0,0 @@ -package options - -import ( - "fmt" - "os" - - "github.com/coder/serpent" -) - -var buildSecretPrefix = fmt.Sprintf("%sBUILD_SECRET_", envPrefix) - -// EnvWithBuildSecretPrefix returns a string in the format of a build secret environment variable. -func EnvWithBuildSecretPrefix(secretName, secretValue string) string { - return fmt.Sprintf("%s%s=%s", buildSecretPrefix, secretName, secretValue) -} - -// GetBuildSecrets sources build secrets from the given environment. -// -// In a normal docker build, build secrets would be passed in via the -// `docker build --secret` flag. envbuilder is more analogous to a -// `docker run` that just happens to build its own container. It doesn't have -// access to the `--secret` flag. As an alternative, we source these from the -// envbuilder process environment. -func GetBuildSecrets(environ []string) []string { - buildSecrets := serpent.ParseEnviron(environ, buildSecretPrefix).ToOS() - return buildSecrets -} - -// ClearBuildSecretsFromProcessEnvironment unsets all build secrets from the process environment. -// NOTE: This does not remove them from /proc/self/environ. They are still visible -// there unless execve(2) is called. -// -// Unlike runtime secrets in the devcontainer spec or orchestration systems like -// Kubernetes, build secrets should not be available at run time. envbuilder blurs -// the line between build time and run time by transitioning from one to the other -// within the same process in the same container. -// -// These build secrets should not make it into the runtime environment of the runtime -// container init process. It is therefore useful to unset build secret environment -// variables to ensure they aren't accidentally passed into the exec call. -func ClearBuildSecretsFromProcessEnvironment() { - buildSecrets := serpent.ParseEnviron(os.Environ(), buildSecretPrefix) - for _, secret := range buildSecrets { - os.Unsetenv(buildSecretPrefix + secret.Name) - } -} diff --git a/options/secrets_test.go b/options/secrets_test.go deleted file mode 100644 index 69193d95..00000000 --- a/options/secrets_test.go +++ /dev/null @@ -1,138 +0,0 @@ -package options_test - -import ( - "os" - "strings" - "testing" - - "github.com/coder/envbuilder/options" - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" -) - -func TestGetBuildSecrets(t *testing.T) { - // This test cannot be run in parallel, because it needs to modify the OS environment - tests := []struct { - name string - envVars map[string]string - expectedSecrets []string - }{ - { - name: "no secrets set", - envVars: map[string]string{}, - expectedSecrets: []string{}, - }, - { - name: "single secret", - envVars: map[string]string{ - "ENVBUILDER_BUILD_SECRET_FOO": "bar", - }, - expectedSecrets: []string{"FOO=bar"}, - }, - { - name: "multiple secrets", - envVars: map[string]string{ - "ENVBUILDER_BUILD_SECRET_FOO": "bar", - "NOT_A_SECRET": "baz", - "ENVBUILDER_BUILD_SECRET_BAZ": "qux", - }, - expectedSecrets: []string{"FOO=bar", "BAZ=qux"}, - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - preserveEnv(t) - os.Clearenv() - - options.ClearBuildSecretsFromProcessEnvironment() - require.Empty(t, options.GetBuildSecrets(os.Environ())) - - // Set environment variables for the test case - for key, value := range tt.envVars { - t.Setenv(key, value) - } - - // when - secrets := options.GetBuildSecrets(os.Environ()) - - // then - assert.ElementsMatch(t, tt.expectedSecrets, secrets) - }) - } -} - -func TestClearBuildSecrets(t *testing.T) { - // This test cannot be run in parallel, because it needs to modify the OS environment - tests := []struct { - name string - initialEnvVars map[string]string - expectedSecretsBeforeClear []string - expectedEnvironAfterClear []string - }{ - { - name: "single secret", - initialEnvVars: map[string]string{ - "ENVBUILDER_BUILD_SECRET_FOO": "bar", - }, - expectedSecretsBeforeClear: []string{"FOO=bar"}, - }, - { - name: "multiple secrets", - initialEnvVars: map[string]string{ - "ENVBUILDER_BUILD_SECRET_FOO": "bar", - "ENVBUILDER_BUILD_SECRET_BAZ": "qux", - }, - expectedSecretsBeforeClear: []string{"FOO=bar", "BAZ=qux"}, - }, - { - name: "only build secrets are cleared", - initialEnvVars: map[string]string{ - "ENVBUILDER_BUILD_SECRET_FOO": "foo", - "BAR": "bar", - }, - expectedSecretsBeforeClear: []string{"FOO=foo"}, - expectedEnvironAfterClear: []string{"BAR=bar"}, - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - preserveEnv(t) - os.Clearenv() - - // Set environment variables for the test case - for key, value := range tt.initialEnvVars { - t.Setenv(key, value) - } - - // Verify secrets before clearing - secrets := options.GetBuildSecrets(os.Environ()) - assert.ElementsMatch(t, tt.expectedSecretsBeforeClear, secrets) - - // Clear the secrets - options.ClearBuildSecretsFromProcessEnvironment() - - // Verify secrets after clearing - environ := os.Environ() - secrets = options.GetBuildSecrets(environ) - assert.Empty(t, secrets) - }) - } -} - -// preserveEnv takes a snapshot of the current process environment and restores it after the current -// test to ensure that we don't cause flakes by modifying the environment for other tests. -func preserveEnv(t *testing.T) { - envSnapshot := make(map[string]string) - for _, envVar := range os.Environ() { - parts := strings.SplitN(envVar, "=", 2) - envSnapshot[parts[0]] = parts[1] - } - t.Cleanup(func() { - os.Clearenv() - for key, value := range envSnapshot { - os.Setenv(key, value) - } - }) -} diff --git a/options/testdata/options.golden b/options/testdata/options.golden index 397d5e0b..840b2340 100644 --- a/options/testdata/options.golden +++ b/options/testdata/options.golden @@ -12,6 +12,10 @@ OPTIONS: WorkspaceFolder. This path MUST be relative to the WorkspaceFolder path into which the repo is cloned. + --build-secrets string-array, $ENVBUILDER_BUILD_SECRETS + The list of secret environment variables to use when building the + image. + --cache-repo string, $ENVBUILDER_CACHE_REPO The name of the container registry to push the cache image to. If this is empty, the cache will not be pushed. From 5b6c62a45391f723c9d6a9e980ecb0e5efb5159f Mon Sep 17 00:00:00 2001 From: Sas Swart Date: Mon, 4 Nov 2024 07:39:59 +0000 Subject: [PATCH 6/9] fix(docs): fix example in the build secret docs --- docs/build-secrets.md | 66 ++++++++++++++++++++++++++++++------------- 1 file changed, 46 insertions(+), 20 deletions(-) diff --git a/docs/build-secrets.md b/docs/build-secrets.md index 5f39c1d0..c3c88900 100644 --- a/docs/build-secrets.md +++ b/docs/build-secrets.md @@ -12,39 +12,60 @@ To illustrate build secrets in Envbuilder, let's build, push and run a container First, start a local docker registry, so that we can push and inspect the built image: ```bash -docker run --rm -d -p 5000:5000 --name Envbuilder-registry registry:2 +docker run --rm -d -p 5000:5000 --name envbuilder-registry registry:2 ``` -Then, build an image based on this Dockerfile: - -```Dockerfile -FROM alpine:latest +Then, prepare the files to build our container. +```bash +mkdir test-build-secrets +cd test-build-secrets +printf 'FROM alpine:latest\n\nRUN --mount=type=secret,id=TEST_BUILD_SECRET_A,env=TEST_BUILD_SECRET_A echo -n $TEST_BUILD_SECRET_A | sha256sum > /foo_secret_hash.txt\nRUN --mount=type=secret,id=TEST_BUILD_SECRET_B,dst=/tmp/bar.secret cat /tmp/bar.secret | sha256sum > /bar_secret_hash.txt\n' > Dockerfile +printf '{"build": { "dockerfile": "Dockerfile"}}\n' > devcontainer.json +``` -RUN --mount=type=secret,id=FOO,env cat $FOO > /foo_secret_hash.txt -RUN --mount=type=secret,id=BAR,dst=/tmp/bar.secret cat /tmp/bar.secret > /bar_secret_hash.txt +Inspect the Dockerfile and devcontainer.json files in the new directory. +```bash +cat devcontainer.json +cat Dockerfile ``` -using this command: + +Note that the Dockerfile requires two secrets: `TEST_BUILD_SECRET_A` and `TEST_BUILD_SECRET_B`. Their values are arbitrarily set to `secret-foo` and `secret-bar` by the command below. Building the container image writes the checksums for these secrets to disk. This illustrates that the secrets can be used in the build to enact side effects without exposing the secrets themselves. + +Execute the build using this command: ```bash docker run -it --rm \ - -e ENVBUILDER_BUILD_SECRETS='FOO=envbuilder-test-secret-foo,BAR=envbuilder-test-secret-bar' \ + -e ENVBUILDER_BUILD_SECRETS='TEST_BUILD_SECRET_A=secret-foo,TEST_BUILD_SECRET_B=secret-bar' \ -e ENVBUILDER_INIT_SCRIPT='/bin/sh' \ - -e ENVBUILDER_CACHE_REPO=$(docker inspect Envbuilder-registry | jq -r '.[].NetworkSettings.IPAddress'):5000/test-container \ + -e ENVBUILDER_CACHE_REPO=$(docker inspect envbuilder-registry | jq -r '.[].NetworkSettings.IPAddress'):5000/test-container \ -e ENVBUILDER_PUSH_IMAGE=1 \ -v $PWD:/workspaces/empty \ - ghcr.io/coder/Envbuilder:latest + ghcr.io/coder/envbuilder:latest ``` This will result in a shell session inside the built container. -You can now verify two things: +You can now verify three things: * The secrets provided to build are not available once the container is running. They are no longer on disk, nor are they in the process environment, or in `/proc/self/environ`. -* The secrets were still useful during the build. The following comnmands show that the secrets had side effects inside the build, without remaining in the image: ```bash +cat /proc/self/environ | tr '\0' '\n' +printenv +``` +* The secrets were still useful during the build. The following commands show that the secrets had side effects inside the build, without remaining in the image: +```bash +echo -n "secret-foo" | sha256sum cat /foo_secret_hash.txt +echo -n "secret-bar" | sha256sum cat /bar_secret_hash.txt ``` +Notice that the first two checksums match and that the last two checksums match. + +Finally, exit the container: +```bash +exit +``` + ### Verifying that images are secret free -To verify that the build image doesn't contain build secrets, run the following: +To verify that the built image doesn't contain build secrets, run the following: ```bash docker pull localhost:5000/test-container:latest @@ -53,19 +74,24 @@ mkdir -p test-container tar -xf test-container.tar -C test-container/ cd test-container # Scan image layers for secrets: -find . -type f | xargs tar -xOf 2>/dev/null | strings | grep -rin "envbuilder-test-secret" +find . -type f | xargs tar -xOf 2>/dev/null | strings | grep -rn "secret-foo" +find . -type f | xargs tar -xOf 2>/dev/null | strings | grep -rn "secret-bar" # Scan image manifests for secrets: -find . -type f | xargs -n1 grep -rinI 'envbuilder-test-secret' +find . -type f | xargs -n1 grep -rnI 'secret-foo' +find . -type f | xargs -n1 grep -rnI 'secret-bar' cd ../ ``` The output of both find/grep commands should be empty. -To verify that it scans correctly, replace "envbuilder-test-secret" with "Envbuilder" and rerun the commands. It should find strings related to Envbuilder that are not secrets. +To verify that it scans correctly, replace "secret-foo" with "envbuilder" and rerun the commands. It should find strings related to Envbuilder that are not secrets. + +### Cleanup -Having verified that no secrets were included in the image, we can now delete the artifacts that we saved to disk. +Having verified that no secrets were included in the image, we can now delete the artifacts that we saved to disk and remove the containers. ```bash -rm -r test-container -rm -r test-container.tar +cd ../ +rm -r test-build-secrets +docker stop envbuilder-registry ``` ## Security and Production Use From b10c6e30110e8a14742eab4fda15847da2fbc49c Mon Sep 17 00:00:00 2001 From: Sas Swart Date: Mon, 4 Nov 2024 07:45:29 +0000 Subject: [PATCH 7/9] fix(docs): update docs to include the new build secrets flag and env --- docs/env-variables.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/env-variables.md b/docs/env-variables.md index cb7054f3..6f9653bd 100644 --- a/docs/env-variables.md +++ b/docs/env-variables.md @@ -21,6 +21,7 @@ | `--force-safe` | `ENVBUILDER_FORCE_SAFE` | | Ignores any filesystem safety checks. This could cause serious harm to your system! This is used in cases where bypass is needed to unblock customers. | | `--insecure` | `ENVBUILDER_INSECURE` | | Bypass TLS verification when cloning and pulling from container registries. | | `--ignore-paths` | `ENVBUILDER_IGNORE_PATHS` | | The comma separated list of paths to ignore when building the workspace. | +| `--build-secrets` | `ENVBUILDER_BUILD_SECRETS` | | The list of secret environment variables to use when building the image. | | `--skip-rebuild` | `ENVBUILDER_SKIP_REBUILD` | | Skip building if the MagicFile exists. This is used to skip building when a container is restarting. e.g. docker stop -> docker start This value can always be set to true - even if the container is being started for the first time. | | `--git-url` | `ENVBUILDER_GIT_URL` | | The URL of a Git repository containing a Devcontainer or Docker image to clone. This is optional. | | `--git-clone-depth` | `ENVBUILDER_GIT_CLONE_DEPTH` | | The depth to use when cloning the Git repository. | From 2fcd340c01430bd1eef4463dceb6b5270ef3817e Mon Sep 17 00:00:00 2001 From: Sas Swart Date: Mon, 4 Nov 2024 10:14:54 +0000 Subject: [PATCH 8/9] fix(docs): update docs on build secrets for legibility --- docs/build-secrets.md | 74 ++++++++++++++++++++++++++++++++++--------- 1 file changed, 59 insertions(+), 15 deletions(-) diff --git a/docs/build-secrets.md b/docs/build-secrets.md index c3c88900..c1b57236 100644 --- a/docs/build-secrets.md +++ b/docs/build-secrets.md @@ -19,17 +19,22 @@ Then, prepare the files to build our container. ```bash mkdir test-build-secrets cd test-build-secrets -printf 'FROM alpine:latest\n\nRUN --mount=type=secret,id=TEST_BUILD_SECRET_A,env=TEST_BUILD_SECRET_A echo -n $TEST_BUILD_SECRET_A | sha256sum > /foo_secret_hash.txt\nRUN --mount=type=secret,id=TEST_BUILD_SECRET_B,dst=/tmp/bar.secret cat /tmp/bar.secret | sha256sum > /bar_secret_hash.txt\n' > Dockerfile -printf '{"build": { "dockerfile": "Dockerfile"}}\n' > devcontainer.json +cat << EOF > Dockerfile +FROM alpine:latest + +RUN --mount=type=secret,id=TEST_BUILD_SECRET_A,env=TEST_BUILD_SECRET_A echo -n \$TEST_BUILD_SECRET_A | sha256sum > /foo_secret_hash.txt +RUN --mount=type=secret,id=TEST_BUILD_SECRET_B,dst=/tmp/bar.secret cat /tmp/bar.secret | sha256sum > /bar_secret_hash.txt +EOF +cat << EOF > devcontainer.json +{ + "build": { + "dockerfile": "Dockerfile" + } +} +EOF ``` -Inspect the Dockerfile and devcontainer.json files in the new directory. -```bash -cat devcontainer.json -cat Dockerfile -``` - -Note that the Dockerfile requires two secrets: `TEST_BUILD_SECRET_A` and `TEST_BUILD_SECRET_B`. Their values are arbitrarily set to `secret-foo` and `secret-bar` by the command below. Building the container image writes the checksums for these secrets to disk. This illustrates that the secrets can be used in the build to enact side effects without exposing the secrets themselves. +The Dockerfile requires two secrets: `TEST_BUILD_SECRET_A` and `TEST_BUILD_SECRET_B`. Their values are arbitrarily set to `secret-foo` and `secret-bar` by the command below. Building the container image writes the checksums for these secrets to disk. This illustrates that the secrets can be used in the build to enact side effects without exposing the secrets themselves. Execute the build using this command: ```bash @@ -43,13 +48,41 @@ docker run -it --rm \ ``` This will result in a shell session inside the built container. -You can now verify three things: -* The secrets provided to build are not available once the container is running. They are no longer on disk, nor are they in the process environment, or in `/proc/self/environ`. +You can now verify two things: + +Firstly, the secrets provided to build are not available once the container is running. They are no longer on disk, nor are they in the process environment, or in `/proc/self/environ`: ```bash cat /proc/self/environ | tr '\0' '\n' printenv ``` -* The secrets were still useful during the build. The following commands show that the secrets had side effects inside the build, without remaining in the image: +Expected output: +```bash +/workspaces/empty # cat /proc/self/environ | tr '\0' '\n' +HOSTNAME=c0b0ee3d5564 +SHLVL=2 +HOME=/root +TERM=xterm +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +DEVCONTAINER_CONFIG=/workspaces/empty/devcontainer.json +ENVBUILDER=true +TS_DEBUG_TRIM_WIREGUARD=false +PWD=/workspaces/empty +DEVCONTAINER=true +/workspaces/empty # printenv +HOSTNAME=c0b0ee3d5564 +SHLVL=2 +HOME=/root +TERM=xterm +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +DEVCONTAINER_CONFIG=/workspaces/empty/devcontainer.json +ENVBUILDER=true +TS_DEBUG_TRIM_WIREGUARD=false +PWD=/workspaces/empty +DEVCONTAINER=true +/workspaces/empty # +``` + +Finally, the secrets were still useful during the build. The following commands show that the secrets had side effects inside the build, without remaining in the image: ```bash echo -n "secret-foo" | sha256sum cat /foo_secret_hash.txt @@ -57,7 +90,18 @@ echo -n "secret-bar" | sha256sum cat /bar_secret_hash.txt ``` -Notice that the first two checksums match and that the last two checksums match. +Notice that the first two checksums match and that the last two checksums match. Expected output: +``` +/workspaces/empty # echo -n "secret-foo" | sha256sum +9a888f08a057159d2ea8fb69d38c9a25e367d7ca3128035b7f6dee2ca988c3d8 - +/workspaces/empty # cat /foo_secret_hash.txt +9a888f08a057159d2ea8fb69d38c9a25e367d7ca3128035b7f6dee2ca988c3d8 - +/workspaces/empty # echo -n "secret-bar" | sha256sum +fb1c9d1220e429b30c60d028b882f735b5af72d7b5496d9202737fe9f1d38289 - +/workspaces/empty # cat /bar_secret_hash.txt +fb1c9d1220e429b30c60d028b882f735b5af72d7b5496d9202737fe9f1d38289 - +/workspaces/empty # +``` Finally, exit the container: ```bash @@ -103,10 +147,10 @@ Build secrets are meant for use cases where the secret should not be accessible Build secrets are only protected if they are not copied or moved from their location as designated in the `RUN` directive. If a build secret is used, care should be taken to ensure that it is not copied or otherwise persisted into an image layer beyond the control of Envbuilder. ### Who should be able to access build secrets, when and where? -The secure way to use build secrets with Envbuilder is to deny users access to the platform that hosts Envbuilder. Only grant access to the Envbuilder container once it has concluded its build, using a trusted non-platform channel like ssh or the coder agent running inside the container. Once control has been handed to such a runtime container process, Envbuilder will have cleared all secrets that it set from the container. - Anyone with sufficient access to attach directly to the container (eg. using `kubectl`), will be able to read build secrets if they attach to the container before it has concluded its build. Anyone with sufficient access to the platform that hosts the Envbuilder container will also be able to read these build secrets from where the platform stores them. This is true for other build systems, and containerised software in general. +The secure way to use build secrets with Envbuilder is to deny users access to the platform that hosts Envbuilder. Only grant access to the Envbuilder container once it has concluded its build, using a trusted non-platform channel like ssh or the coder agent running inside the container. Once control has been handed to such a runtime container process, Envbuilder will have cleared all secrets that it set from the container. + If secrets should be accessible at runtime, do not use build secrets. Rather, mount the secret data using a volume or environment variable. Envbuilder will not include mounted volumes in the image that it pushes to any cache repositories, but they will still be available to users that connect to the container. ### Container Management beyond Envbuilder's control From 03bd8b85bdf58e8143c44a821ec301ede4854a68 Mon Sep 17 00:00:00 2001 From: Sas Swart Date: Mon, 4 Nov 2024 10:40:13 +0000 Subject: [PATCH 9/9] fix(docs): update docs on build secrets to illustrate volume safety --- docs/build-secrets.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/docs/build-secrets.md b/docs/build-secrets.md index c1b57236..00c38c7c 100644 --- a/docs/build-secrets.md +++ b/docs/build-secrets.md @@ -32,9 +32,10 @@ cat << EOF > devcontainer.json } } EOF +echo 'runtime-secret-a' > runtime-secret.txt ``` -The Dockerfile requires two secrets: `TEST_BUILD_SECRET_A` and `TEST_BUILD_SECRET_B`. Their values are arbitrarily set to `secret-foo` and `secret-bar` by the command below. Building the container image writes the checksums for these secrets to disk. This illustrates that the secrets can be used in the build to enact side effects without exposing the secrets themselves. +The Dockerfile requires two build secrets: `TEST_BUILD_SECRET_A` and `TEST_BUILD_SECRET_B`. Their values are arbitrarily set to `secret-foo` and `secret-bar` by the command below. Building the container image writes the checksums for these secrets to disk. This illustrates that the secrets can be used in the build to enact side effects without exposing the secrets themselves. Execute the build using this command: ```bash @@ -44,11 +45,12 @@ docker run -it --rm \ -e ENVBUILDER_CACHE_REPO=$(docker inspect envbuilder-registry | jq -r '.[].NetworkSettings.IPAddress'):5000/test-container \ -e ENVBUILDER_PUSH_IMAGE=1 \ -v $PWD:/workspaces/empty \ + -v $PWD/runtime-secret.txt:/runtime-secret.txt \ ghcr.io/coder/envbuilder:latest ``` This will result in a shell session inside the built container. -You can now verify two things: +You can now verify three things: Firstly, the secrets provided to build are not available once the container is running. They are no longer on disk, nor are they in the process environment, or in `/proc/self/environ`: ```bash @@ -82,7 +84,7 @@ DEVCONTAINER=true /workspaces/empty # ``` -Finally, the secrets were still useful during the build. The following commands show that the secrets had side effects inside the build, without remaining in the image: +Secondly, the secrets were still useful during the build. The following commands show that the secrets had side effects inside the build, without remaining in the image: ```bash echo -n "secret-foo" | sha256sum cat /foo_secret_hash.txt @@ -103,6 +105,8 @@ fb1c9d1220e429b30c60d028b882f735b5af72d7b5496d9202737fe9f1d38289 - /workspaces/empty # ``` +Thirdly, the runtime secret that was mounted as a volume is still mounted into the container and accessible. This is why volumes are inappropriate analogues to native docker build secrets. However, notice further down that this runtime secret volume's contents are not present in the built image. It is therefore safe to mount a volume into envbuilder for use during runtime without fear that it will be present in the image that envbuilder builds. + Finally, exit the container: ```bash exit @@ -120,13 +124,15 @@ cd test-container # Scan image layers for secrets: find . -type f | xargs tar -xOf 2>/dev/null | strings | grep -rn "secret-foo" find . -type f | xargs tar -xOf 2>/dev/null | strings | grep -rn "secret-bar" +find . -type f | xargs tar -xOf 2>/dev/null | strings | grep -rn "runtime-secret" # Scan image manifests for secrets: find . -type f | xargs -n1 grep -rnI 'secret-foo' find . -type f | xargs -n1 grep -rnI 'secret-bar' +find . -type f | xargs -n1 grep -rnI 'runtime-secret' cd ../ ``` -The output of both find/grep commands should be empty. +The output of all find/grep commands should be empty. To verify that it scans correctly, replace "secret-foo" with "envbuilder" and rerun the commands. It should find strings related to Envbuilder that are not secrets. ### Cleanup