| title | authors | reviewers | creation-date | last-updated | status | |||||
|---|---|---|---|---|---|---|---|---|---|---|
Creating Control Plane Components |
|
|
2020-10-26 |
2020-11-09 |
provisional |
- Creating Control Plane Components
Refer to the CAPN Glossary.
The goal of this proposal is to define CRDs of the three major components (kube-apiserver~(KAS), Etcd, kube-controller-manager~(KCM)) of the NCP, and a standard process of creating them.
CAPN aims at providing control plane level isolation while sharing physical resources among control planes. There exist various approaches to creating isolated control planes. For example, one can run components of the nested control plane as pods on the underlying clusters, create NCPs through cloud providers' Kubernetes services or use out-of-tree component controllers to create each component. In this proposal, we try to define CRDs of the NCP's three major components and a standard process of creating the three components regardless of which underlying approach is used. As examples, we introduce two setups that 1) creating each component natively, 2) creating KAM and KCM natively while using the Etcd-cluster-operator to create the Etcd.
-
Define the CRD that represents each control plane component. The CRD needs to meet two requirements:
- Portable - the CRD should hold information that is required by different component controllers, e.g., etcdadm, etcd-operator, and etcd-cluster-operator
- Customizable - the CRD should allow end-users to customize each component, i.e., specify the image, component version, and command-line options.
-
Define a standard process of creating control plane components for NCP.
-
Support independently creating/updating each component
-
Define how NCP controller works.
-
Discuss the implementation details of the out-of-tree component controllers.
Generally, creating the three major components requires similar high-level information, like the components' version, the number of replicas, and the amount of computing resources. Meanwhile, end-users should be able to customize NCP components, i.e., specifying the component image, version, and command-line options. Therefore, we define a new struct NestedComponentSpec that contains common information required by different providers as well as customized information specified by the end-users. The NestedComponentSpec will look like the following
type NestedComponentSpec struct {
// NestedComponentSpec defines the common information for creating the component
// +optional
addonv1alpha1.CommonSpec `json:",inline"`
// PatchSpecs includes the user specifed settings
// +optional
addonv1alpha1.PatchSpec `json:",inline"`
// Resources defines the amount of computing resources that will be used by this component
// +optional
Resources corev1.ResourceRequirements `json:"resources",omitempty`
// Replicas defines the number of replicas in the component's workload
// +optional
Replicas int32 `json:"replicas",omitempty`
}The CommonSpecs and the PatchSpec are defined in kubebuilder-declarative-pattern:
// CommonSpec defines the set of configuration attributes that must be exposed on all addons.
type CommonSpec struct {
// Version specifies the exact addon version to be deployed, eg 1.2.3
// It should not be specified if Channel is specified
Version string `json:"version,omitempty"`
// Channel specifies a channel that can be used to resolve a specific addon, eg: stable
// It will be ignored if Version is specified
Channel string `json:"channel,omitempty"`
}
// +k8s:deepcopy-gen=true
type PatchSpec struct {
Patches []*runtime.RawExtension `json:"patches,omitempty"`
}We assume that the APIServer, ContollerManager, Etcd, and the NCP CR are located in the same namespace. To create an NCP, we need first to create NestedAPIserver CR, NestedControllerManager CR, NestedEtcd CR, NCP CR, and a namespace that holds all the CRs, then the component controller can cooperate to create components for the NCP.
As there exist dependencies between components, i.e., KAS cannot run without Etcd, KCM cannot work without KAS, when creating NCP components, component controllers will need to get information and status of other CRs. To achieve this, we will add three ObjectReference to the NestedControlPlaneSpec with each ObjectReference points to a component.
type NestedControlPlaneSpec struct {
// other fields ...
// EtcdRef is the eference to the NestedEtcd
EtcdRef *corev1.ObjectReference `json:"etcd,omitempty"`
// APIServerRef is the reference to the NestedAPIServer
APIServerRef *corev1.ObjectReference `json:"apiserver,omitempty"`
// ContollerManagerRef is the reference to the NestedControllerManager
ControllerManagerRef *corev1.ObjectReference `json:"controllerManager,omitempty"`
}After applying the NCP CR, the NCP controller will find the three associated components and set their metav1.OwnerReference as the NCP CR. Then, the component controller can find other CRs through the owner NCP, when creating the corresponding component workload.
End-users can create component CRs manually and apply them to the cluster with an NCP to create the resources. In the future, we might introduce the Template CR, which will handle the creation of the component CRs in it's controller. We assume that there will be only one component controller for each component at any given time, and it is the cluster administrator's responsibility to set up the proper component controllers.
The component controller will create the component under the in-tree mode, which will create the component using the default manifests. The readiness and liveness probe will be used, and we will mark each component as ready only when the corresponding workload is ready. As the KAS cannot work without available Etcd and the KCM cannot run without KAS, the three components need to be created by their respective controllers in the order of Etcd, KAS, and KCM. Creation order is maintained using cross resource status, which checks and wait until the dependencies are provisioned. We will host sets of default templates in this repository. Users can specify which set of templates they intend to use by specifying the corresponding version or channel in the embedded CommonSpec in the component's CR.
Each component's controller will generate necessary certificates for the component and store them to the secret resources defined by CAPI. Also, The KAS controller will store the content of the kubeconfig file in a secret named [clustername]-kubeconfig.
The creating process will include six steps:
-
The user generates all CRs, i.e., NCP, Etcd, APIServer, ControllerManager, with the same namespace, and apply them.
-
The Etcd controller generates the certificates (including a root CA and TLS serving certificates), creates the Etcd workload, and stores the certificates into
secret/[cluster-name]-etcd -
The KAS controller creates a KAS service (for exposing the NCP), generates certificates (including a root CA, TLS serving certificates), creates the KAS workload, stores the certificates into
secret/[cluster-name]-ca, creates a kubeconfig and stores it intosecret/[cluster-name]-kubeconfig. -
The KCM controller generates the KCM kubeconfig and creates the KCM workload.
-
After all the three components are ready, the NCP controller marks the NCP CR as ready.
If users intend to use an external controller to create the NCP component, they may need to implement a new component controller that can interact with the component CR and the external controller to create the component. For example, if the user wanted to use the etcd-cluster-operator that requires the EtcdCluster CR. They need to implement a custom controller that watches the NestedEtcd resource, creates the necessary CRs for that implementation, and updates the required status fields on NestedEtcd to allow dependent services to be provisioned. This can be done using the kubebuilder-declarative-pattern like is done for in-tree component controllers.
In the following example, we assume that the user intends to use Etcd-cluster-operator(ECO) as the Etcd controller. The creating process will include seven steps:
-
The cluster administrator deletes the in-tree Etcd controller and deploys the custom Etcd controller (ECO controller).
-
The user generates all CRs and apply them.
-
The ECO controller creates the EtcdCluster CR.
-
The ECO creates the Etcd workload.
-
At the meantime, the ECO controller keeps watching the EtcdCluster CR, stores the Etcd CA into the
secret/[cluster-name]-etcd, and updates theEtcdCR accordingly. -
The KAS controller creates the KAS service, generates certificates, creates the KAS workload, stores certificates into
secret/[cluster-name]-ca, creates the kubeconfig and stores it into thesecret/[cluster-name]-kubeconfig -
The KCM controller generates the KCM kubeconfig and creates the KCM workload.
-
Once all the three components are ready, the NCP controller marks the NCP CR as ready.
The followings are CRDs of the three components.
// NestedEtcdSpec defines the desired state of Etcd
type NestedEtcdSpec struct {
// NestedComponentSpec contains the common and user-specified information that are
// required for creating the component
// +optional
NestedComponentSpec `json:",inline"`
}
// NestedEtcdStatus defines the observed state of Etcd
type NestedEtcdStatus struct {
// Ready is set if all resources have been created
Ready bool `json:"ready,omitempty"`
// Addresses defines how to address the etcd instance
Addresses []NestedEtcdAddress `json:"addresses,omitempty"`
// CommonStatus allows addons status monitoring
addonv1alpha1.CommonStatus `json:",inline"`
}
// EtcdAddress defines the observed addresses for etcd
type NestedEtcdAddress struct {
// IP Address of the etcd instance.
// +optional
IP string `json:"ip,omitempty"`
// Hostname of the etcd instance
Hostname string `json:"hostname,omitempty"`
// Port of the etcd instance
// +optional
Port int32 `json:"port"`
}
// NestedEtcd is the Schema for the Etcd API
type NestedEtcd struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec EtcdSpec `json:"spec,omitempty"`
Status EtcdStatus `json:"status,omitempty"`
}type NestedAPIServerSpec struct {
// NestedComponentSpec contains the common and user-specified information that are
// required for creating the component
// +optional
NestedComponentSpec `json:",inline"`
}
// NestedAPIServerStatus defines the observed state of APIServer
type NestedAPIServerStatus struct {
// Ready is set if all resources have been created
// +kubebuilder:default=false
Ready bool `json:"ready,omitempty"`
// APIServerService is the reference to the service that expose the APIServer
// +optional
APIServerService *corev1.ObjectReference `json:"apiserverService,omitempty"`
// CommonStatus allows addons status monitoring
addonv1alpha1. CommonStatus `json:",inline"`
}
// NestedAPIServer is the Schema for the APIServers API
type NestedAPIServer struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec NestedAPIServerSpec `json:"spec,omitempty"`
Status NestedAPIServerStatus `json:"status,omitempty"`
}// NestedControllerManagerSpec defines the desired state of ControllerManager
type NestedControllerManagerSec struct {
// NestedComponentSpec contains the common and user-specified information that are
// required for creating the component
// +optional
NestedComponentSpec `json:",inline"`
}
// NestedControllerManagerStatus defines the observed state of ControllerManager
type NestedControllerManagerStatus struct {
// Ready is set if all resources have been created
Ready bool `json:"ready,omitempty"`
// CommonStatus allows addons status monitoring
addonv1alpha1. CommonStatus `json:",inline"`
}
// NestedControllerManager is the Schema for the ControllerManagers API
type NestedControllerManager struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec NestedControllerManagerSpec `json:"spec,omitempty"`
Status NestedControllerManagerStatus `json:"status,omitempty"`
}Creating an NCP requires the end-user to submit a creation request, and the cluster administrator will be responsible for creating the NCP CRs and applying them. Once the NCP is ready, the cluster administrator will return a kubeconfig to the end-user, as each end-user can only access the apiserver assigned to them. There is no need to worry about malicious users to manipulate other users' resources. A malicious user can still skew the system by creating a massive amount of resources. To avoid this, we need to enhance the syncer component; however, this topic is beyond this proposal's scope. The proposed mechanism will not lead to any severe security issues.
- MM/DD/YYYY: Proposed idea in an issue or [community meeting]
- MM/DD/YYYY: Compile a Google Doc following the CAEP template (link here)
- MM/DD/YYYY: First round of feedback from community
- MM/DD/YYYY: Present proposal at a [community meeting]
- MM/DD/YYYY: Open proposal PR