Allow AZs to be Omitted at Runtime

CAPI appears to make implicit CP scheduling decisions based on what it's
told is available by CAPO on an "LRU" basis. It also assumes an infinite
sized AZ, so problems begin when the "next" AZ cannot accommodate the
VM. We could manually specify an AZ that aggregates all machines
explcitly, however this is another mechanism that disables scheduling by
CAPI altogether and allows Nova to do what it does, along with a soft-AA
rule. However, switching from CAPI scheduling to Nova scheduling is
impossible as the field is immutable, so allow this.  Testing shows
existing scheduled clusters undergo no topology changes, which will be
due to the KCPM not taking action, but you can force the changes with a
rolling upgrade of some variety. Crucially, if a cluster with CAPI
scheduling gets stuck, we can modify to Nova scheduling and it should
pick up the new specification and get past the hurdle.

Author: spjmurray

api/v1alpha8/openstackcluster_webhook.go

@@ -130,6 +130,11 @@ func (r *OpenStackCluster) ValidateUpdate(oldRaw runtime.Object) (admission.Warn
 	old.Spec.ControlPlaneAvailabilityZones = []string{}
 	r.Spec.ControlPlaneAvailabilityZones = []string{}
 
+	// Allow the scheduling to be changed from CAPI managed to Nova and
+	// vice versa.
+	old.Spec.ControlPlaneOmitAvailabilityZone = false
+	r.Spec.ControlPlaneOmitAvailabilityZone = false
+
 	// Allow change to the allowAllInClusterTraffic.
 	old.Spec.AllowAllInClusterTraffic = false
 	r.Spec.AllowAllInClusterTraffic = false

api/v1alpha8/openstackcluster_webhook_test.go

@@ -234,6 +234,21 @@ func TestOpenStackCluster_ValidateUpdate(t *testing.T) {
 			},
 			wantErr: false,
 		},
+		{
+			name: "Modifying OpenstackCluster.Spec.ControlPlaneOmitAvailabilityZone is allowed",
+			oldTemplate: &OpenStackCluster{
+				Spec: OpenStackClusterSpec{
+					CloudName: "foobar",
+				},
+			},
+			newTemplate: &OpenStackCluster{
+				Spec: OpenStackClusterSpec{
+					CloudName:                        "foobar",
+					ControlPlaneOmitAvailabilityZone: true,
+				},
+			},
+			wantErr: false,
+		},
 		{
 			name: "Changing OpenStackCluster.Spec.APIServerFixedIP is allowed when API Server Floating IP is disabled",
 			oldTemplate: &OpenStackCluster{