build CAPO supported images

[tormath1]

/kind feature

Describe the solution you'd like:

During the CAPO office hours, @mdbooth suggested to build CAPO supported images in the CI. These images could be directly consumed by the tests (and consumed by the users?).

These images could be built for multiple Kubernetes version and multiple OS version (ubuntu-2004, flatcar, etc.)

Anything else you would like to add:

On the implementation side, we could use Github Actions and image-builder Docker image. Example with Flatcar:

sudo docker run --privileged --net host -v "${PWD}/docker-output:/output" -v /dev:/dev --rm \
    -e PACKER_FLAGS="--var 'kubernetes_semver=v1.26.2' --var 'kubernetes_series=v1.26'" \
    -e OEM_ID=openstack \
    tormath1/cluster-node-image-builder-amd64 \
    build-qemu-flatcar

(Note: it needs kubernetes-sigs/image-builder#1092)

For Flatcar: flatcar/Flatcar#928 (kubernetes-sigs/image-builder#924) can be interesting

Produced images could be then stored on a GCS bucket.

[tormath1]

Initial experiments: tormath1@d981cfb but limited because Github Actions do not support KVM (except on hosted runners)

[mdbooth]

@tormath1 We would probably use cloudbuild rather than GH actions, and I think that might support KVM.

[tormath1]

No more luck with cloudbuild - using N1_HIGHCPU_8 which should support nested virtualization.

EDIT: Trying now with no acceleration, just to see how many times it would take.

[tormath1]

Here's the result with no acceleration: https://github.com/tormath1/cluster-api-provider-openstack/actions/runs/4408341205/usage - generated artifacts are currently uploaded on Github but we could send them to a GCS bucket.

[mdbooth]

That's awesome!

I think we're going to need this in cloudbuild for 2 reasons:

• We need the artifacts in a GCS bucket for CI or we'll exceed our ingress quota.
• The only automated way (caveat: that I'm aware of) to use the project's gcloud credentials is from the prow secure cluster.

That said, I don't currently know when I or anybody else will find time to implement this and you seem to have something looks close to useful for at least some people already. I wonder if we should merge a GH actions solution now and aim to replace it with a cloudbuild solution in prow when we get to it.

Thoughts?
/cc @lentzi90 @tobiasgiese @jichenjc

[tormath1]

Moving it to cloudbuild is not an issue as it mainly relies on the Makefile and Docker to run. It's just less community friendly as one can easily browse the logs on the Github actions (or maybe we can have the logs from cloubuild ? I don't know how it works :D)

[mdbooth]

Moving it to cloudbuild is not an issue as it mainly relies on the Makefile and Docker to run. It's just less community friendly as one can easily browse the logs on the Github actions (or maybe we can have the logs from cloubuild ? I don't know how it works :D)

Yeah, we have the logs: they're in the prow job that runs the cloudbuild, same as regular tests. Admittedly they're not as easy to find, though. This feels like an opportunity for improved docs 🤔

[mdbooth]

For reference, here's the logs of one of our nightly image builds: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/cluster-api-provider-openstack-push-images-nightly/1635189053307490304

Incidentally, that job seems pointless. We should probably kill it.

[mdbooth]

Example of automatic uploading of artifacts from a cloudbuild job:

cluster-api-provider-openstack/Makefile

Lines 405 to 406 in b2b2a3e

	upload-staging-artifacts: ## Upload release artifacts to the staging bucket
	gsutil cp $(RELEASE_DIR)/* gs://$(STAGING_BUCKET)/components/$(RELEASE_ALIAS_TAG)/

[tormath1]

Ported the Github Action to cloudbuild, it works fine with a single job but not when running in parallel the four builds (flatcar x2 + ubuntu x2). It might be some network issue with Packer trying to SSH... (logs)

[mdbooth]

Could it just be timeouts? It's probably fine to run them sequentially as separate cloudbuild steps, I guess.

[tormath1]

I don't think it's timeout, it's very early in the builds - we can run them sequentially if 6h of build is not an issue :)