EKS cluster AWSMachines not created because MachineSet pre-fight check fails

[tingchienc]

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

Add a MachineDeployment for the EKS cluster deployed by CAPA
No errors in the capa-controller pods

But there is an error for the machineDeployment.

Error messages:
Message: Performing "Scale up" on hold because ControlPlane namespace/cluster-control-plane is provisioning ("ControlPlaneIsStable" preflight failed). The operation will continue after the preflight check(s) pass

After we set the annotation "machineset.cluster.x-k8s.io/skip-preflight-checks: ControlPlaneIsStable", the EC2 machines finally created.

The problem is why the preflight-checks fail? We don't want to add the annotation for every MachineDeployment.
And it seems like a new feature added in Sep by ClusterAPI.

What did you expect to happen:
EC2 machines created without adding the annotation

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api-provider-aws version: latest
• Kubernetes version: (use kubectl version): 1.29.0
• OS (e.g. from /etc/os-release): ubuntu

[k8s-ci-robot]

This issue is currently awaiting triage.

If CAPA/CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[oneironautjack]

Experiencing the same issue but using existing VPC without EKS. Did you find a permanent solution or are you still using the workaround?

[mfrehe]

Hello,

i am facing the same issue with EKS.

After some research i recognized that the MachineSetPreflightCheck was introduced as an experimental feature in CAPI version 1.5 and has been promoted to beta status in CAPI version 1.9, where it is enabled by default (kubernetes-sigs/cluster-api#11117).

So i'd assume that you are using CAPI >= v1.9.0 (like me) which seems to be not supported by CAPA v2.7.1

An issue for CAPI v1.9.0 was created in Nov 24: #5218

As i am afraid of downgrading to CAPI v1.8.x (which version seems to be supported by CAPA, #5061), i'd like to ask for a CAPA version that supports CAPI >= v1.9.0.

Edit:
Maybe caused by the introduction of v1beta2 structs (kubernetes-sigs/cluster-api#11234)?

Regards
Marcel

[oneironautjack]

Any luck with the downgrades? I've attempted to downgrade CAPI to v1.8.6 to no avail.

[dkoshkin]

This is the check code https://github.com/kubernetes-sigs/cluster-api/blob/79e6731b0d20a0ca2dad5e4f96d2a532aa82b54b/internal/contract/controlplane.go#L190-L208

It is expecting status.version on AWSManagedControlPlane.

---

https://cluster-api.sigs.k8s.io/developer/providers/contracts/control-plane#controlplane-version

In case you are developing a control plane provider which allows control of the version of the Kubernetes control plane instances in your control plane, following fields MUST be implemented in the ControlPlane

In this case since AWSManagedControlPlane is used for both controlPlaneRef and infrastructureRef I think it is not adhering to the contract, see API.

[damdo]

Thanks for reporting this.
We had a discussion about this on Slack this morning with the Core CAPI maintainers.
See it here: https://kubernetes.slack.com/archives/CD6U2V71N/p1739783013734149

TLDR: we are planning to update the status field for AWSManagedControlPlane to include Version, and make the CAPI Pre Flight checks happy.
I am working on it here: #5209