feat: properly support kube-vip upgrades #1062

dkoshkin · 2025-02-28T00:25:22Z

What problem does this PR solve?:
See https://docs.google.com/document/d/1RolIuLXf5qD5l2lya0ZtY-ZcGkh07ctiRiG8jjXMthU/edit?usp=sharing with additional details.

We ran into an issue where upgrading CAREN causes an unwanted rollout of all control plane Nodes.
This happens when CAREN is shipped with a new kube-vip version and the difference in the static Pod spec changes the file contents in KCP and forces new Nodes to be created.

We do not want this behavior and only want this rollout to happen at controlled times.

In a recent change we introduced a mechanism of making backend changes by using versioned handlers. 7bc9094

This change follows a similar pattern. A new handler is introduced that will instead of reading the kube-vip template from a global ConfigMap will read it directly from the KCPTemplate associated with the ClusterClass.
We can then continue to change the kube-vip version (or any other spec changes there) and downstream projects will just pick up whatever changes through a new ClusterClass.

Another important goal with this approach was to not require consumers of CAREN to maintain some reference to the "correct" kube-vip template. We want to keep upgrades as simple as possible by only changing the Kubernetes version, and let CAREN handle its own business logic.

Alternative Considered Solutions

Add a ConfigMap ref of the static Pod template to the API. This would require the clients to keep track of and update another field on upgrades. We want to keep the upgrade changes as simple as possible and leave this kind of implementation detail to CAREN.
Add logic to only apply new changes when upgrading. This approach was considered for a previous scenario, but we decided against it due to complexity and instead introduced versioned ClusterClass and templates.

Which issue(s) this PR fixes:
Fixes #

How Has This Been Tested?:

Tested locally by creating a cluster with the new and old handlers.

Special notes for your reviewer:

This PR introduces a "v3" version of the handlers with the modified kube-vip handler, and preserves the previous behavior from v2.

pkg/handlers/v2/generic/mutation/controlplanevirtualip/inject.go

pkg/handlers/generic/mutation/controlplanevirtualip/providers/kubevip_test.go

pkg/handlers/nutanix/mutation/controlplanevirtualip/inject.go

pkg/handlers/generic/mutation/controlplanevirtualip/inject_test.go

pkg/handlers/generic/mutation/controlplanevirtualip/inject.go

pkg/handlers/generic/mutation/controlplanevirtualip/inject_test.go

pkg/handlers/generic/mutation/controlplanevirtualip/providers/kubevip.go

pkg/handlers/v2/generic/mutation/controlplanevirtualip/inject.go

thunderboltsid

LGTM overall; left a few comments. It'd have been easier for review if v2 and delete could also have been two separate commits (browser kept dying while reviewing) :D

To correctly support kube-vip version upgrades without an unplanned control-plane rollout, read the kube-vip template directly from the KCPTemplate that will be released along with the ClusterClasses. # Conflicts: # pkg/handlers/generic/mutation/controlplanevirtualip/providers/kubevip_test.go

hack/examples/bases/nutanix/clusterclass/kustomization.yaml.tmpl

pkg/handlers/generic/mutation/controlplanevirtualip/inject.go

supershal

Thanks the solution and suggesting alternatives.

jimmidyson

This is great!

@jimmidyson

🤖 I have created a release *beep* *boop* --- ## 0.28.0 (2025-04-10)  ## What's Changed ### Exciting New Features 🎉 * feat: Update all addon versions by @jimmidyson in #1054 * feat: go 1.24.1 and update all other tools by @jimmidyson in #1066 * feat: bumped frr-routing to 9.1.3 by @ArvinderPal09 in #1067 * feat: properly support kube-vip upgrades by @dkoshkin in #1062 * feat: update addons by @dkoshkin in #1072 * feat: enable Cilium's hubble relay mTLS by @dkoshkin in #1086 * feat: adds image tempalting for capx by @faiq in #1096 * feat: support setting kubeadm ignorePreflightErrors by @dkoshkin in #1097 * feat: go 1.24.2 to fix CVE by @jimmidyson in #1100 * feat: update Calico to v3.29.3 by @dkoshkin in #1101 ### Fixes 🔧 * fix: correctly handle multiple registries with the same Host by @dkoshkin in #1063 * fix: Tolerate all NoSchedule taints for NFD and CSI deployments by @jimmidyson in #1074 * fix: Retain existing join and init kubeadm config when adding taints by @jimmidyson in #1073 ### Other Changes * build: Update k8s versions for tests by @jimmidyson in #1052 * docs: updates cilium doc with link to its default spec by @manoj-nutanix in #1036 * ci: Replace deprecated blackduck scan action by @jimmidyson in #1070 * build: Update k8s.io/{kubelet,utils} deps by @jimmidyson in #1078 * revert: "build: Update k8s.io/{kubelet,utils} deps" by @jimmidyson in #1080 * ci: Enable dependabot for hack/tools module by @jimmidyson in #1082 * build: Update k8s.io/kubelet dep by @jimmidyson in #1081 * ci: Use Kubernetes minor in e2e check name by @jimmidyson in #1090 ## New Contributors * @ArvinderPal09 made their first contribution in #1067 **Full Changelog**: v0.27.1...v0.28.0 --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Dimitri Koshkin <[email protected]>

dkoshkin requested review from jimmidyson, thunderboltsid and supershal February 28, 2025 00:25

github-actions bot added the feature label Feb 28, 2025

dkoshkin force-pushed the dkoshkin/feat-properly-support-kube-vip-upgrades branch from f207d9a to 1ab9c31 Compare February 28, 2025 00:29

dkoshkin commented Feb 28, 2025

View reviewed changes

pkg/handlers/v2/generic/mutation/controlplanevirtualip/inject.go Show resolved Hide resolved

dkoshkin force-pushed the dkoshkin/feat-properly-support-kube-vip-upgrades branch 2 times, most recently from 3385406 to 3bfe1e1 Compare February 28, 2025 01:10

dkoshkin commented Feb 28, 2025

View reviewed changes

pkg/handlers/generic/mutation/controlplanevirtualip/providers/kubevip_test.go Show resolved Hide resolved

dkoshkin commented Feb 28, 2025

View reviewed changes

pkg/handlers/nutanix/mutation/controlplanevirtualip/inject.go Show resolved Hide resolved

dkoshkin commented Feb 28, 2025

View reviewed changes

pkg/handlers/generic/mutation/controlplanevirtualip/inject_test.go Show resolved Hide resolved

github-actions bot added feature and removed feature labels Feb 28, 2025