feat: implement reproducible build and get cached image #213

mafredri · 2024-05-30T14:28:23Z

This PR is a PoC, it builds upon #197 and coder/kaniko#12 and shows how build cache + pushed image can be utilized.

Example get-cached-image:

❯ registry_ip=$(docker inspect envbuilder-registry | jq -r '.[].NetworkSettings.IPAddress'); docker run -it --rm -v /tmp/envbuilder:/workspaces -e ENVBUILDER_CACHE_REPO=${registry_ip}:5000/local/cache -e ENVBUILDER_GIT_URL=https://github.com/coder/envbuilder-starter-devcontainer -e ENVBUILDER_INIT_SCRIPT=exit -e ENVBUILDER_GET_CACHED_IMAGE=true envbuilder:latest
envbuilder - Build development environments from repositories in a container
#1: 📦 Cloning https://github.com/coder/envbuilder-starter-devcontainer to /workspaces/envbuilder-starter-devcontainer...
...
2: RUN apt-get update
2: unable to build fake image: error fake building stage: uncached command *commands.RunMarkerCommand is not supported in fake build

Now, do build:

❯ registry_ip=$(docker inspect envbuilder-registry | jq -r '.[].NetworkSettings.IPAddress'); docker run -it --rm -v /tmp/envbuilder:/workspaces -e ENVBUILDER_CACHE_REPO=${registry_ip}:5000/local/cache -e ENVBUILDER_GIT_URL=https://github.com/coder/envbuilder-starter-devcontainer -e ENVBUILDER_INIT_SCRIPT=exit -e ENVBUILDER_PUSH_IMAGE=true envbuilder:latest
envbuilder - Build development environments from repositories in a container
#1: 📦 Cloning https://github.com/coder/envbuilder-starter-devcontainer to /workspaces/envbuilder-starter-devcontainer...
...
2: Running: [/bin/sh -c echo "PS1='🐳 \[\033[1;36m\]\h \[\033[1;34m\]\W\[\033[0;35m\] \[\033[1;36m\]# \[\033[0m\]'" > ~/.bashrc]
2: Taking snapshot of files...
2: Pushing layer 172.17.0.3:5000/local/cache:445691a7ed1a9ce9faac28095b3f3b77a0043c7604b099c299bd4abed5089f57 to cache now
2: Pushing image to 172.17.0.3:5000/local/cache:445691a7ed1a9ce9faac28095b3f3b77a0043c7604b099c299bd4abed5089f57
2: Pushed 172.17.0.3:5000/local/cache@sha256:440a1a035eb3ba7597aa7b3f1b5e08c67c3c5afed33bea057dd18640420ca115
#2: 🏗️ Built image! [16.509477016s]
#3: 🏗️ Pushing image...
3: Pushing image to 172.17.0.3:5000/local/cache
3: Pushed 172.17.0.3:5000/local/cache@sha256:9c102cd02585011898abd11ea367ba5368415b9cd9ecebbb07dcbb57f6e4cd63
#3: 🏗️ Pushed image! [1.003674439s]
=== Running the init command /bin/sh [-c exit] as the "coder" user...

Try get-cached-image again:

❯ registry_ip=$(docker inspect envbuilder-registry | jq -r '.[].NetworkSettings.IPAddress'); docker run -it --rm -v /tmp/envbuilder:/workspaces -e ENVBUILDER_CACHE_REPO=${registry_ip}:5000/local/cache -e ENVBUILDER_GIT_URL=https://github.com/coder/envbuilder-starter-devcontainer -e ENVBUILDER_INIT_SCRIPT=exit -e ENVBUILDER_GET_CACHED_IMAGE=true envbuilder:latest
envbuilder - Build development environments from repositories in a container
#1: 📦 Cloning https://github.com/coder/envbuilder-starter-devcontainer to /workspaces/envbuilder-starter-devcontainer...
...
#2: 🏗️ Building fake image...
2: Retrieving image manifest ubuntu
2: Retrieving image ubuntu from registry index.docker.io
2: Built cross stage deps: map[]
2: Retrieving image manifest ubuntu
2: Returning cached image manifest
2: Executing 0 build triggers
2: Checking for cached layer 172.17.0.3:5000/local/cache:bd0def545b590c39144a57152c120a5baf8e3cad7c70298bbacfa5116ef1ca3c...
2: Using caching version of cmd: RUN apt-get update
2: Checking for cached layer 172.17.0.3:5000/local/cache:591f8c35a55cc60f7bc49fa3f2c99360616a9b6b35908b8aaa7392e7820b0c36...
2: Using caching version of cmd: RUN apt-get install vim sudo curl -y
2: Checking for cached layer 172.17.0.3:5000/local/cache:1cb1d8b00537df6f0638913717931b76e7b80b8c150448ce4096f4eef042b355...
2: Using caching version of cmd: RUN useradd -m -s /bin/bash -G sudo coder
2: Cmd: USER
2: Checking for cached layer 172.17.0.3:5000/local/cache:445691a7ed1a9ce9faac28095b3f3b77a0043c7604b099c299bd4abed5089f57...
2: Using caching version of cmd: RUN echo "PS1='🐳 \[\033[1;36m\]\h \[\033[1;34m\]\W\[\033[0;35m\] \[\033[1;36m\]# \[\033[0m\]'" > ~/.bashrc
2: RUN apt-get update
2: Found cached layer, faking extraction to filesystem
2: RUN apt-get install vim sudo curl -y
2: Found cached layer, faking extraction to filesystem
2: RUN useradd -m -s /bin/bash -G sudo coder
2: Found cached layer, faking extraction to filesystem
2: USER coder
2: Cmd: USER
2: No files changed in this command, skipping snapshotting.
2: RUN echo "PS1='🐳 \[\033[1;36m\]\h \[\033[1;34m\]\W\[\033[0;35m\] \[\033[1;36m\]# \[\033[0m\]'" > ~/.bashrc
2: Found cached layer, faking extraction to filesystem
2: Mapping stage idx 0 to digest sha256:3ff463dd27a13ebcdfc74fc790e7a8da0b5373110b4edd1dc1426c4b4ed80e27
2: Mapping digest sha256:3ff463dd27a13ebcdfc74fc790e7a8da0b5373110b4edd1dc1426c4b4ed80e27 to cachekey 445691a7ed1a9ce9faac28095b3f3b77a0043c7604b099c299bd4abed5089f57
#2: 🏗️ Built fake image! [1.298919769s]
172.17.0.3:5000/local/cache@sha256:9c102cd02585011898abd11ea367ba5368415b9cd9ecebbb07dcbb57f6e4cd63

And finally:

❯ docker run -it --rm --entrypoint /usr/bin/bash localhost:5000/local/cache@sha256:9c102cd02585011898abd11ea367ba5368415b9cd9ecebbb07dcbb57f6e4cd63
Unable to find image 'localhost:5000/local/cache@sha256:9c102cd02585011898abd11ea367ba5368415b9cd9ecebbb07dcbb57f6e4cd63' locally
localhost:5000/local/cache@sha256:9c102cd02585011898abd11ea367ba5368415b9cd9ecebbb07dcbb57f6e4cd63: Pulling from local/cache
49b384cc7b4a: Already exists
4e5c7f007bcd: Pull complete
b3cda1764760: Pull complete
62220e490978: Pull complete
07141b433862: Pull complete
Digest: sha256:9c102cd02585011898abd11ea367ba5368415b9cd9ecebbb07dcbb57f6e4cd63
Status: Downloaded newer image for localhost:5000/local/cache@sha256:9c102cd02585011898abd11ea367ba5368415b9cd9ecebbb07dcbb57f6e4cd63
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

🐳 e54cf491d1a7 / #

Closes #186.

This commit enables pushing the final image to the given cache repo.a As part of #185, the goal is to allow for faster startup when the image has already been built.

options.go

johnstcn · 2024-05-31T09:32:58Z

envbuilder.go

 			// Boilerplate!
 			CustomPlatform:    platforms.Format(platforms.Normalize(platforms.DefaultSpec())),
 			SnapshotMode:      "redo",
 			RunV2:             true,
 			RunStdout:         stdoutWriter,
 			RunStderr:         stderrWriter,
-			Destinations:      []string{"local"},
+			Destinations:      destinations,
+			NoPush:            !options.PushImage || len(destinations) == 0,


I'm assuming the reason for the logical disjuction here is that Kaniko will push all intermediate cache layers to destinations, while specifying options.PushImage signifies to push only the final result?

Yep 👍🏻

envbuilder.go

BrunoQuaresma · 2024-06-07T16:09:59Z

envbuilder.go

-		})
+
+			// For cached image utilization, produce reproducible builds.
+			Reproducible: options.PushImage,


Where is this getting used?

See coder/kaniko#12

kylecarbs · 2024-06-11T12:58:08Z

I'd love to try this in dogfood asap with our devcontainer!

kylecarbs · 2024-06-11T12:58:27Z

And since we added support for sysbox, this might just work!

johnstcn · 2024-06-11T13:00:28Z

I'd love to try this in dogfood asap with our devcontainer!

Samesies! Do you think we'd want a registry on each dogfood host or just one shared registry?

kylecarbs · 2024-06-11T13:04:16Z

I think we try using the Docker Registry to see what the performance looks like with a remote!

johnstcn · 2024-06-11T14:25:51Z

envbuilder.go

+	if options.CacheRepo == "" && options.PushImage {
+		return fmt.Errorf("--cache-repo must be set when using --push-image!")
+	}


review: I don't see any good reason to not do this. It prevents potential head-scratching if someone sets ENVBUILDER_PUSH_IMAGE without specifying ENVBUILDER_CACHE_REPO.

dry run maybe?

johnstcn · 2024-06-11T16:38:32Z

@dannykopping @BrunoQuaresma @mtojek I have updated this branch with integration tests.

It's worth noting that --get-cached-image currently does not work with multi-stage builds. I have filed a separate issue for that.

mtojek

We agreed on DoCacheProbe in kaniko's PR, so this round is fairly quick. Approved 👍

README.md

mtojek · 2024-06-12T06:58:12Z

envbuilder.go

+	if options.CacheRepo == "" && options.PushImage {
+		return fmt.Errorf("--cache-repo must be set when using --push-image!")
+	}


dry run maybe?

dannykopping

Only nits, great work!

dannykopping · 2024-06-12T06:52:46Z

integration/integration_test.go

+	})
+
+	t.Run("CacheAndPushMultistage", func(t *testing.T) {
+		// Currently fails with:


Is there a way we can make this fail more gracefully? This output is rather difficult to reason about.

There probably is, but I'd say we should handle it as part of #230

envbuilder.go

integration/integration_test.go

options.go

Co-authored-by: Danny Kopping <[email protected]>

mafredri added 3 commits May 30, 2024 14:03

feat: push final image to cache repo

d0b4d6a

This commit enables pushing the final image to the given cache repo.a As part of #185, the goal is to allow for faster startup when the image has already been built.

explicitly set nopush if no destinations

5a50f4a

wrap

6cb1217

mafredri requested review from dannykopping, johnstcn and kylecarbs May 30, 2024 14:34

mafredri added 2 commits May 30, 2024 14:55

feat: implement reproducible build and get cached image

7764535

gen

d54ca81

mafredri force-pushed the mafredri/feat-fake-build branch from 42b6121 to d54ca81 Compare May 30, 2024 15:56

mafredri mentioned this pull request May 30, 2024

Add support for fake, and speed up reproducible builds coder/kaniko#12

Merged

gomod

ea91cd5

johnstcn assigned mafredri May 31, 2024

johnstcn reviewed May 31, 2024

View reviewed changes

This was referenced Jun 1, 2024

PoC: Build a tool to validate file hashes against layer caches #186

Closed

Write RFC around performance/caching improvements #128

Closed

Embed the envbuilder binary in images #216

Closed

Sanitize the final container image for envbuilder #217

Closed

mtojek assigned johnstcn Jun 3, 2024

mtojek mentioned this pull request Jun 3, 2024

Envbuilder v1.0 release #132

Closed

36 tasks

BrunoQuaresma reviewed Jun 7, 2024

View reviewed changes

johnstcn mentioned this pull request Jun 11, 2024

feat: push final image to cache repo #197

Closed

chore: add unit tests for push image / cache mode

1979ffd

johnstcn reviewed Jun 11, 2024

View reviewed changes

johnstcn added 3 commits June 11, 2024 15:26

unused code

a962a26

update options.golden

d804fc9

Merge branch 'main' into mafredri/feat-fake-build

3b18839

add test for multistage build

b7502ee

mtojek self-requested a review June 11, 2024 16:53

mtojek approved these changes Jun 12, 2024

View reviewed changes

dannykopping approved these changes Jun 12, 2024

View reviewed changes

johnstcn and others added 4 commits June 12, 2024 09:38

Apply suggestions from code review

aacdbfc

Co-authored-by: Danny Kopping <[email protected]>

fix help output

149bdb3

update kaniko to remove fake verbiage when probing build cache

354d885

Merge branch 'main' into mafredri/feat-fake-build

15904df

johnstcn merged commit eb01b08 into main Jun 12, 2024
4 checks passed

johnstcn deleted the mafredri/feat-fake-build branch June 12, 2024 10:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement reproducible build and get cached image #213

feat: implement reproducible build and get cached image #213

mafredri commented May 30, 2024 •

edited

Loading

johnstcn May 31, 2024

mafredri May 31, 2024

BrunoQuaresma Jun 7, 2024

johnstcn Jun 10, 2024

kylecarbs commented Jun 11, 2024

kylecarbs commented Jun 11, 2024

johnstcn commented Jun 11, 2024

kylecarbs commented Jun 11, 2024

johnstcn Jun 11, 2024

mtojek Jun 12, 2024

johnstcn commented Jun 11, 2024

mtojek left a comment

mtojek Jun 12, 2024

dannykopping left a comment

dannykopping Jun 12, 2024

johnstcn Jun 12, 2024

feat: implement reproducible build and get cached image #213

feat: implement reproducible build and get cached image #213

Conversation

mafredri commented May 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kylecarbs commented Jun 11, 2024

kylecarbs commented Jun 11, 2024

johnstcn commented Jun 11, 2024

kylecarbs commented Jun 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnstcn commented Jun 11, 2024

mtojek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dannykopping left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mafredri commented May 30, 2024 •

edited

Loading