Skip to content

chore: update dind examples to use onCreateCommand #350

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Sep 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 16 additions & 9 deletions docs/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ from inside Envbuilder.
> you may need to instead add the relevant content of the init script to your
> agent startup script in your template.
> For example:
> ```
>
> ```terraform
> resource "coder_agent" "dev" {
> ...
> startup_script = <<-EOT
Expand Down Expand Up @@ -43,7 +44,6 @@ docker run -it --rm \
ghcr.io/coder/envbuilder:latest
```


## Docker-in-Docker (DinD)

**Security:** Low
Expand All @@ -57,16 +57,16 @@ Example:

> Note that due to a lack of init system, the Docker daemon
> needs to be started separately inside the container. In this example, we
> create a custom entrypoint to start the Docker daemon in the background and
> call this entrypoint via `ENVBUILDER_INIT_SCRIPT`.
> create a custom script to start the Docker daemon in the background and
> call this entrypoint via the Devcontainer `onCreateCommand` lifecycle hook.

```console
docker run -it --rm \
--privileged \
-v /tmp/envbuilder:/workspaces \
-e ENVBUILDER_GIT_URL=https://github.com/coder/envbuilder \
-e ENVBUILDER_DEVCONTAINER_DIR=/workspaces/envbuilder/examples/docker/02_dind \
-e ENVBUILDER_INIT_SCRIPT=/entrypoint.sh \
-e ENVBUILDER_INIT_SCRIPT=bash \
ghcr.io/coder/envbuilder:latest
```

Expand All @@ -75,8 +75,14 @@ docker run -it --rm \
The above can also be accomplished using the [`docker-in-docker` Devcontainer
feature](https://github.com/devcontainers/features/tree/main/src/docker-in-docker).

> Note: we still need the custom entrypoint to start the docker startup script.
> See https://github.com/devcontainers/features/blob/main/src/docker-in-docker/devcontainer-feature.json#L60
> Note: we still need the `onCreateCommand` to start Docker.
> See
> [here](https://github.com/devcontainers/features/blob/main/src/docker-in-docker/devcontainer-feature.json#L65)
> for more details.
>
> Known issue: `/run` does not get symlinked correctly to `/var/run`.
> To work around this, we create the symlink manually before running
> the script to start the Docker daemon.

Example:

Expand All @@ -86,7 +92,7 @@ docker run -it --rm \
-v /tmp/envbuilder:/workspaces \
-e ENVBUILDER_GIT_URL=https://github.com/coder/envbuilder \
-e ENVBUILDER_DEVCONTAINER_DIR=/workspaces/envbuilder/examples/docker/03_dind_feature \
-e ENVBUILDER_INIT_SCRIPT=/entrypoint.sh \
-e ENVBUILDER_INIT_SCRIPT=bash \
ghcr.io/coder/envbuilder:latest
```

Expand All @@ -95,7 +101,7 @@ docker run -it --rm \
**Security:** Medium
**Convenience:** Medium

This approach runs a Docker daemon in *rootless* mode.
This approach runs a Docker daemon in _rootless_ mode.
While this still requires a privileged container, this allows you to restrict
usage of the `root` user inside the container, as the Docker daemon will be run
under a "fake" root user (via `rootlesskit`). The user inside the workspace can
Expand Down Expand Up @@ -129,6 +135,7 @@ including transparently enabling Docker inside workspaces. Most notably, it
access inside their workspaces, if required.

Example:

```console
docker run -it --rm \
-v /tmp/envbuilder:/workspaces \
Expand Down
25 changes: 21 additions & 4 deletions examples/docker/02_dind/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,23 @@
FROM ubuntu:noble

# Install Docker using Docker's convenience script.
RUN apt-get update && \
apt-get install -y curl apt-transport-https && \
curl -fsSL https://get.docker.com/ | sh -s -
ADD entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
apt-get install -y curl sudo apt-transport-https && \
curl -fsSL https://get.docker.com/ | sh -s -

# The ubuntu:noble image includes a non-root user by default,
# but it does not have sudo privileges. We need to set this up.
# Note: we chown /var/run/docker.sock to the non-root user
# in the onCreateCommand script. Ideally you would add the
# non-root user to the docker group, but in this scenario
# this is a 'single-user' environment. It also avoids us
# having to run `newgrp docker`.
RUN echo "ubuntu ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/ubuntu

# Add our onCreateCommand script.
ADD on-create.sh /on-create.sh

# Switch to the non-root user.
USER ubuntu

ENTRYPOINT ["bash"]
5 changes: 3 additions & 2 deletions examples/docker/02_dind/devcontainer.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
{
"build": {
"dockerfile": "Dockerfile"
}
}
},
"onCreateCommand": "/on-create.sh"
}
7 changes: 0 additions & 7 deletions examples/docker/02_dind/entrypoint.sh

This file was deleted.

22 changes: 22 additions & 0 deletions examples/docker/02_dind/on-create.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/usr/bin/env bash

set -euo pipefail

# Start Docker in the background.
sudo -u root /bin/sh -c 'nohup dockerd > /var/log/docker.log &'

# Wait up to 10 seconds for Docker to start.
for attempt in $(seq 1 10); do
if [[ $attempt -eq 10 ]]; then
echo "Failed to start Docker"
exit 1
fi
if [[ ! -e /var/run/docker.sock ]]; then
sleep 1
else
break
fi
done

# Change the owner of the Docker socket so that the non-root user can use it.
sudo chown ubuntu:docker /var/run/docker.sock
23 changes: 21 additions & 2 deletions examples/docker/03_dind_feature/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,22 @@
FROM ubuntu:noble
ADD entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

# Install some dependencies such as curl and sudo.
# Also set up passwordless sudo for the ubuntu user.
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get install -y \
curl \
sudo \
apt-transport-https && \
echo "ubuntu ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/ubuntu

# Add our onCreateCommand script.
ADD on-create.sh /on-create.sh

# Switch to the non-root user.
USER ubuntu

# The devcontainer feature provides /usr/local/share/docker-init.sh
# which will handle most of the steps of setting up Docker.
# We can't put this in the entrypoint as it gets overridden, so
# we call it in the on-create script.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we not support setting entrypoint in the Dockerfile at all? Maybe we should in the future.

Copy link
Member Author

@johnstcn johnstcn Sep 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My observation is that whatever entrypoint you set gets overridden by ENVBUILDER_INIT_COMMAND / ENVBUILDER_INIT_SCRIPT. But agreed, we should fall back to the entrypoint in the Dockerfile.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For a Coder workspace we need to have coder_agent.*.init_script as the entry point so another option to handle the ENTRYPOINT is to run it as part of coder_agent startup_script. This will work for at least the case when envbuilder is used with Coder.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filed #351 to follow up on this.

ENTRYPOINT ["bash"]
3 changes: 2 additions & 1 deletion examples/docker/03_dind_feature/devcontainer.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@
"build": {
"dockerfile": "Dockerfile"
},
"onCreateCommand": "/on-create.sh",
"features": {
"ghcr.io/devcontainers/features/docker-in-docker:2": {}
}
}
}
7 changes: 0 additions & 7 deletions examples/docker/03_dind_feature/entrypoint.sh

This file was deleted.

18 changes: 18 additions & 0 deletions examples/docker/03_dind_feature/on-create.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
#!/usr/bin/env bash

set -euo pipefail

# Known issue: Kaniko does not symlink /run => /var/run properly.
# This results in /var/run/ being owned by root:root which interferes
# with accessing the Docker socket even if the permissions are set
# correctly. Workaround: symlink it manually
sudo ln -s /run /var/run

# Run the docker init script. This needs to be
# run as root. It will take care of starting the
# daemon and adding the ubuntu user to the docker
# group.
sudo /usr/local/share/docker-init.sh

# Change the owner of the Docker socket so that the non-root user can use it.
sudo chown ubuntu:docker /var/run/docker.sock
11 changes: 8 additions & 3 deletions examples/docker/04_dind_rootless/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
FROM ubuntu:noble

# Based on UID of ubuntu user in container.
ENV XDG_RUNTIME_DIR /run/user/1000
ENV DOCKER_HOST unix:///${XDG_RUNTIME_DIR}/docker.sock

# Setup as root
USER root
RUN apt-get update && \
# Install prerequisites
apt-get install -y apt-transport-https curl iproute2 uidmap && \
Expand All @@ -19,6 +22,8 @@ USER ubuntu
RUN dockerd-rootless-setuptool.sh install && \
docker context use rootless && \
mkdir -p /home/ubuntu/.local/share/docker
# Add our custom entrypoint
ADD entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

# Add our onCreateCommand script.
ADD on-create.sh /on-create.sh

ENTRYPOINT ["bash"]
5 changes: 3 additions & 2 deletions examples/docker/04_dind_rootless/devcontainer.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
{
"build": {
"dockerfile": "Dockerfile"
}
}
},
"onCreateCommand": "/on-create.sh"
}
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,4 @@
set -euo pipefail

# Start the rootless docker daemon as a non-root user
nohup rootlesskit --net=slirp4netns --mtu=1500 --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run dockerd > "/tmp/dockerd-rootless.log" 2>&1 &

exec bash --login
nohup rootlesskit --net=slirp4netns --mtu=1500 --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run dockerd >"/tmp/dockerd-rootless.log" 2>&1 &