feat: add cluster-autoscaler CRS addon

[dkoshkin]

Adding the cluster-autoscaler addon, starting with ClusterResourceSet first.

The cluster-autoscaler addon is different than the other addons because it needs access to both the CAPI objects and the workload Cluster. Because of this, it needs to always be deployed on the management Cluster and use the CAPI kubeconfig Secret to watch the workload Cluster.

Tested locally, but was not able to work on e2e tests yet, since there is no mechanism yet to make the Cluster self-managed.

[dkoshkin]

Would love some foeedback on this before going much further 🙏

[jimmidyson]

More discussion on cluster-autoscaler and ClusterClass in kubernetes-sigs/cluster-api#8217

[dkoshkin]

More discussion on cluster-autoscaler and ClusterClass in kubernetes-sigs/cluster-api#8217

Interesting! We solved this in DKP by having the CLI client update both the annotations AND the MachineDeployment's replicas with an imperative command.

But I think the --enforce-node-group-min-size flag is a better solution here.

Not a huge fan that the internal annotation details are exposed to the user, but maybe thats ok for the initial implementation.

[jimmidyson]

Nice work!

[jimmidyson]

CRS is not applied from bootstrap to workload:

- lastTransitionTime: "2024-03-07T14:34:49Z"
    message: '[creating object /v1, Kind=ServiceAccount quick-start-zu4cbq/cluster-autoscaler-quick-start-la8a9u:
      namespaces "quick-start-zu4cbq" not found, creating object policy/v1, Kind=PodDisruptionBudget
      quick-start-zu4cbq/cluster-autoscaler-quick-start-la8a9u: namespaces "quick-start-zu4cbq"
      not found, creating object rbac.authorization.k8s.io/v1, Kind=Role quick-start-zu4cbq/cluster-autoscaler-quick-start-la8a9u:
      namespaces "quick-start-zu4cbq" not found, creating object rbac.authorization.k8s.io/v1,
      Kind=RoleBinding quick-start-zu4cbq/cluster-autoscaler-quick-start-la8a9u: namespaces
      "quick-start-zu4cbq" not found, creating object /v1, Kind=Service quick-start-zu4cbq/cluster-autoscaler-quick-start-la8a9u:
      namespaces "quick-start-zu4cbq" not found, creating object apps/v1, Kind=Deployment
      quick-start-zu4cbq/cluster-autoscaler-quick-start-la8a9u: namespaces "quick-start-zu4cbq"
      not found]'

[jimmidyson]

Need to specify the namespace for CA and create all resources in there.

[jimmidyson]

With this approach, the CA will always be deployed on the CP nodes because workers will be set to 0, CA will be deployed and only then will worker nodes be scaled up.

[dkoshkin]

Need to specify the namespace for CA and create all resources in there.

Yep, thank you e2e tests, should be fixed in e9c7342
When I tested locally I created the management cluster in the default namespace so missed that.

[dkoshkin]

With this approach, the CA will always be deployed on the CP nodes because workers will be set to 0, CA will be deployed and only then will worker nodes be scaled up.

The CAPI webhook will set the MD replicas based on the CA annotation https://github.com/kubernetes-sigs/cluster-api/blob/main/internal/webhooks/machinedeployment.go#L289-L291

I also see this behavior running locally when setting --worker-machine-count=3

k get md
NAME                              CLUSTER                REPLICAS   READY   UPDATED   UNAVAILABLE   PHASE     AGE   VERSION
dkoshkin-ca-test-200-md-0-zzxt7   dkoshkin-ca-test-200   3          3       3         0             Running   19m   v1.28.7

We may want to support clusters with 0 replicas, but I think we should do that as a separate effort since other addons would need similar tolerations.

[dkoshkin]

Oops, I didn't see the auto-merge was enabled but the e2e tests passed 🎉