Skip to content

Add a container cleanup job to the linux builders #3814

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
finestructure opened this issue May 31, 2025 · 1 comment
Open

Add a container cleanup job to the linux builders #3814

finestructure opened this issue May 31, 2025 · 1 comment

Comments

@finestructure
Copy link
Member

We've had multiple occasions during the Wasm and Android processing where "stalled" containers blocked processing:

 docker-user@bastion  lb1  ~  docker ps                                                                                                                                                                                      Sat May 31 09:46:36 2025
CONTAINER ID   IMAGE                                                             COMMAND                  CREATED             STATUS             PORTS     NAMES
5252dab38279   registry.gitlab.com/finestructure/spi-images:basic-6.1-latest     "swift build --tripl…"   About an hour ago   Up About an hour             interesting_hamilton
991a521491bf   registry.gitlab.com/finestructure/spi-images:android-6.1-latest   "swift build --swift…"   2 hours ago         Up 2 hours                   musing_hypatia
3e1558adbc95   registry.gitlab.com/finestructure/spi-images:android-6.1-latest   "swift build --swift…"   2 hours ago         Up 2 hours                   recursing_ishizaka
a8a0b51382c2   registry.gitlab.com/finestructure/spi-images:wasm-6.1-latest      "swift build --swift…"   2 hours ago         Up 2 hours                   angry_wilbur

We should have a job running (perhaps as part of the builder itself) that force removes containers older than 15 mins. The only thing we need to be careful about is not to remove deployment containers, as these have a longer timeout. The IMAGE is probably something we'll want to look at for that purpose.

@finestructure
Copy link
Member Author

Adding this to the builder is tempting but potentially problematic: If the linux builder has as many stalled builders as it is wide, there won't be another builder spawned to clean them up.

This will probably need to be a separate process running on each builder.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant