Description
/kind feature
Describe the solution you'd like:
Support of creation network infrastructure, subnets, and gateways, required for AWS
Local Zones and Wavelength zones, allowing users to create compute nodes (non-control
plane) specifying those subnets.
AWS Local Zones allow end-users to deploy cloud resources closer to the metropolitan
regions and offer a limited resource, like Instance Type, Block Storage, Application Load
Balancers, Nat Gateways etc.
The network infrastructure required to create nodes in Local Zones are:
- public and private subnets,
- public subnet associated with a route table with Internet Gateway as a default router
- private subnet associated with a route table with a valid Nat Gateway*
- *Currently, Nat Gateway is not globally supported by Local Zone locations, and currently there
is not API to query if it is offered in the zone.
AWS Wavelength zones are infrastructure running in carrier infrastructure. It also have
limitations of resources offered in those locations.
Network resources required for Wavelength zones:
- Carrier Gateway attached to the VPC
- public and private subnets
- public "carrier" route table with the default route entry targeting the Carrier Gateway
- private route table with the default route entry pointing to a valid Nat Gateway*
- public subnet associated with the public "carrier" route table
- private subnet associated with the private route table
- *Same Nat GW limitation as Local Zone.
Not Goal:
- Create control plane nodes, or required resources, in remote zones.
- Compute/Machine resource creation.
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Currently, OpenShift creates the network resources required to deploy only compute
nodes in AWS Local Zones and Wavelength zones, and we would like to use Cluster API to
deploy those resources. Machines (compute nodes) are then
created, outside Cluster API.
Considering the limitation of network-based load balancers, we used to recommend
to the users to use the AWS Application Load Balancer controller to
create ingress traffic into those zones (day-2).
For the gateway for private subnets, we are reusing the same Nat Gateway for the
Parent Zone (zone attribute), when it exists, otherwise, the
"default" private route table will be used to associate the zone in the remote location.
If possible, it would be nice to deploy Nat Gateway in the location, when supported
(currently, only a few Local Zone locations support it), otherwise, use some strategy like that.
For the gateway for public subnets in Wavelength Zones, we are creating a single
"public carrier" route table, with the default route entry to the Carrier Gateway,
previously created and associated with the VPC,
For more details feel free to read the enhancement with more details here.
The suggested Local Zones workflow looks like this:
- User creates the Subnets (spec.Network.Subnets) specifying all required zones to
create the cluster, including Local Zone or Wavelength Zones, in the existing API
fieldSubnetSpec.AvailabilityZone- keeping the valid placeholderIDto
teach Cluster API to create subnets.
zones := []string{"us-east-1a", "us-east-1b", "us-east-1c", "us-east-1-nyc-1a"}
for _, zone := range zones {
cluster.Spec.NetworkSpec.Subnets = append(cluster.Spec.NetworkSpec.Subnets, capa.SubnetSpec{
AvailabilityZone: zone,
CidrBlock: privateCIDRs[idxCIDR].String(),
ID: fmt.Sprintf("%s-subnet-private-%s", infraID, zone),
IsPublic: false,
})
cluster.Spec.NetworkSpec.Subnets = append(cluster.Spec.NetworkSpec.Subnets, capa.SubnetSpec{
AvailabilityZone: zone,
CidrBlock: publicCIDRs[idxCIDR].String(),
ID: fmt.Sprintf("%s-subnet-public-%s", infraID, zone),
IsPublic: true,
})
idxCIDR++
}- Network reconciliation creates the Carrier Gateway, attaching to the VPC
- Subnet reconciliation creates the public and private subnets into those zones,
then discovers the zone attributes* for the subnet's zone - Subnets prevent returning subnets created in Wavelength Zone locations
to the regular flow, preventing issues in the controllers consuming the standard flow - Route reconciliation check whether the public subnet "is Wavelength Zone", and find the
appropriate route and gateway for public subnets in Carrier Gateways - Route reconciliation check whether the subnet "is Wavelength Zone", and find the
appropriate gateway for private subnets - Regular cluster creation flow
*Required API changes for the proposed workflow:
- Add the following fields to the
SubnetSpec:
// ZoneType defines a zone type for this subnet.
ZoneType *string `json:"zoneType,omitempty"`
// ParentZoneName defines a parent zone name for the zone that the subnet is created,
// when applied. Available only in zone types local-zone or wavelength-zone.
ParentZoneName *string `json:"parentZoneName,omitempty"`The suggested Wavelength Zones workflow looks like this:
- User creates the Subnets (spec.Network.Subnets) specifying all required zones to
create the cluster, including Local Zone or Wavelength Zones, in the existing API
fieldSubnetSpec.AvailabilityZone- keeping the valid placeholderIDto
teach Cluster API to create subnets.
zones := []string{"us-east-1a", "us-east-1b", "us-east-1c", "us-east-1-wl1-nyc-wlz-1"}
...- Subnet reconciliation creates the public and private subnets into those zones,
then discovers the zone attributes* for the subnet's zone - Subnets prevent returning subnets created in Wavelength Zone locations
to the regular flow, preventing issues in the controllers consuming the standard flow - Route reconciliation check whether the subnet "is Local Zone", and find the
appropriate gateway for private subnets - Regular cluster creation flow
*Required API changes for the proposed workflow:
- Add the following fields to the
VPCSpec:
// CarrierGatewayID is the id of the Carrier internet gateway associated with the VPC.
// +optional
CarrierGatewayID *string `json:"carrierGatewayId,omitempty"`### Tasks
- [ ] https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/4882
- [ ] https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/4901
- [ ] 🌱 Implement e2e tests for edge subnets