Skip to content
This repository was archived by the owner on Oct 28, 2024. It is now read-only.

Commit 36a663a

Browse files
charleszheng44charleszheng44
committed
add proposal of creating nested control plane components
1 parent 5f9d4f5 commit 36a663a

File tree

3 files changed

+319
-0
lines changed

3 files changed

+319
-0
lines changed

‎proposals/20201026-creating-control-plane-components.md

+319
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,319 @@
1+
---
2+
title: Creating Control Plane Components
3+
authors:
4+
- "@charleszheng44"
5+
reviewers:
6+
- "@christopherhein"
7+
- "@Fei-Guo"
8+
- "@vincepri"
9+
- "@brightzheng100"
10+
creation-date: 2020-10-26
11+
last-updated: 2020-11-09
12+
status: provisional
13+
---
14+
15+
# Creating Control Plane Components
16+
17+
## Table of Contents
18+
19+
* [Creating Control Plane Components](#creating-control-plane-components)
20+
* [Table of Contents](#table-of-contents)
21+
* [Glossary](#glossary)
22+
* [Summary](#summary)
23+
* [Motivation](#motivation)
24+
* [Goals](#goals)
25+
* [Non-Goals](#non-goals)
26+
* [Proposal](#proposal)
27+
* [Portability and Customizability](#portability-and-customizability)
28+
* [Bootstrap](#bootstrap)
29+
* [Create prerequisites](#create-prerequisites)
30+
* [Creation](#creation)
31+
* [In-tree](#in-tree)
32+
* [Using out-of-tree provisioners](#using-out-of-tree-provisioners)
33+
* [Control Plane Custom Resources](#control-plane-custom-resources)
34+
* [NestedEtcd CRD](#nestedetcd-crd)
35+
* [NestedAPIServer CRD](#nestedapiserver-crd)
36+
* [NestedControllerManager CRD](#nestedcontrollermanager-crd)
37+
* [Security Model](#security-model)
38+
* [Implementation History](#implementation-history)
39+
40+
## Glossary
41+
42+
Refer to the [CAPN Glossary](https://github.com/kubernetes-sigs/cluster-api-provider-nested/blob/master/proposals/00_capn-glossary.md).
43+
44+
## Summary
45+
46+
The goal of this proposal is to define CRDs of the three major components (kube-apiserver~(KAS), Etcd, kube-controller-manager~(KCM)) of the NCP, and a standard process of creating them.
47+
48+
## Motivation
49+
50+
CAPN aims at providing control plane level isolation while sharing physical resources among control planes. There exist various approaches to creating isolated control planes. For example, one can run components of the nested control plane as pods on the underlying clusters, create NCPs through cloud providers' Kubernetes services or use out-of-tree component controllers to create each component. In this proposal, we try to define CRDs of the NCP's three major components and a standard process of creating the three components regardless of which underlying approach is used. As examples, we introduce two setups that 1) creating each component natively, 2) creating KAM and KCM natively while using the [Etcd-cluster-operator](https://github.com/improbable-eng/etcd-cluster-operator) to create the Etcd.
51+
52+
### Goals
53+
54+
- Define the CRD that represents each control plane component. The CRD needs to meet two requirements:
55+
* Portable - the CRD should hold information that is required by different component controllers, e.g., [etcdadm](https://github.com/kubernetes-sigs/etcdadm), [etcd-operator](https://github.com/coreos/etcd-operator), and [etcd-cluster-operator](https://github.com/improbable-eng/etcd-cluster-operator/blob/f84abc6561735814debd67d45bb62d2d2ed8cf4a/api/v1alpha1/etcdcluster_types.go#L31-L47)
56+
* Customizable - the CRD should allow end-users to customize each component, i.e., specify the image, component version, and command-line options.
57+
58+
- Define a standard process of creating control plane components for NCP.
59+
60+
- Support independently creating/updating each component
61+
62+
### Non-Goals
63+
64+
- Define how NCP controller works.
65+
66+
- Discuss the implementation details of the out-of-tree component controllers.
67+
68+
## Proposal
69+
70+
### Portability and Customizability
71+
72+
Generally, creating the three major components requires similar high-level information, like the components' version, the number of replicas, and the amount of computing resources. Meanwhile, end-users should be able to customize NCP components, i.e., specifying the component image, version, and command-line options. Therefore, we define a new struct `NestedComponentSpec` that contains common information required by different providers as well as customized information specified by the end-users. The `NestedComponentSpec` will look like the following
73+
74+
```go
75+
type NestedComponentSpec struct {
76+
// NestedComponentSpec defines the common information for creating the component
77+
// +optional
78+
addonv1alpha1.CommonSpec `json:",inline"`
79+
80+
// PatchSpecs includes the user specifed settings
81+
// +optional
82+
addonv1alpha1.PatchSpec `json:",inline"`
83+
84+
// Resources defines the amount of computing resources that will be used by this component
85+
// +optional
86+
Resources corev1.ResourceRequirements `json:"resources",omitempty`
87+
88+
// Replicas defines the number of replicas in the component's workload
89+
// +optional
90+
Replicas int32 `json:"replicas",omitempty`
91+
}
92+
```
93+
94+
The `CommonSpecs` and the `PatchSpec` are defined in [kubebuilder-declarative-pattern](https://github.com/kubernetes-sigs/kubebuilder-declarative-pattern/blob/1cbf859290cab81ae8e73fc5caebe792280175d1/pkg/patterns/addon/pkg/apis/v1alpha1/common_types.go):
95+
96+
97+
```go
98+
// CommonSpec defines the set of configuration attributes that must be exposed on all addons.
99+
type CommonSpec struct {
100+
// Version specifies the exact addon version to be deployed, eg 1.2.3
101+
// It should not be specified if Channel is specified
102+
Version string `json:"version,omitempty"`
103+
// Channel specifies a channel that can be used to resolve a specific addon, eg: stable
104+
// It will be ignored if Version is specified
105+
Channel string `json:"channel,omitempty"`
106+
}
107+
108+
// +k8s:deepcopy-gen=true
109+
type PatchSpec struct {
110+
Patches []*runtime.RawExtension `json:"patches,omitempty"`
111+
}
112+
```
113+
114+
### Bootstrap
115+
116+
#### Create prerequisites
117+
118+
We assume that the APIServer, ContollerManager, Etcd, and the NCP CR are located in the same namespace. To create an NCP, we need first to create NestedAPIserver CR, NestedControllerManager CR, NestedEtcd CR, NCP CR, and a namespace that holds all the CRs, then the component controller can cooperate to create components for the NCP.
119+
120+
As there exist dependencies between components, i.e., KAS cannot run without Etcd, KCM cannot work without KAS, when creating NCP components, component controllers will need to get information and status of other CRs. To achieve this, we will add three `ObjectReference` to the `NestedControlPlaneSpec` with each `ObjectReference` points to a component.
121+
122+
```go
123+
type NestedControlPlaneSpec struct {
124+
// other fields ...
125+
126+
// EtcdRef is the eference to the NestedEtcd
127+
EtcdRef *corev1.ObjectReference `json:"etcd,omitempty"`
128+
129+
// APIServerRef is the reference to the NestedAPIServer
130+
APIServerRef *corev1.ObjectReference `json:"apiserver,omitempty"`
131+
132+
// ContollerManagerRef is the reference to the NestedControllerManager
133+
ControllerManagerRef *corev1.ObjectReference `json:"controllerManager,omitempty"`
134+
}
135+
```
136+
137+
After applying the NCP CR, the NCP controller will find the three associated components and set their `metav1.OwnerReference` as the NCP CR. Then, the component controller can find other CRs through the owner NCP, when creating the corresponding component workload.
138+
139+
### Creation
140+
141+
End-users can create component CRs manually and apply them to the cluster with an NCP to create the resources. In the future, we might introduce the `Template` CR, which will handle the creation of the component CRs in it's controller. We assume that there will be only one component controller for each component at any given time, and it is the cluster administrator's responsibility to set up the proper component controllers.
142+
143+
#### In-tree
144+
145+
The component controller will create the component under the in-tree mode, which will create the component using the default manifests. The readiness and liveness probe will be used, and we will mark each component as ready only when the corresponding workload is ready. As the KAS cannot work without available Etcd and the KCM cannot run without KAS, the three components need to be created by their respective controllers in the order of Etcd, KAS, and KCM. Creation order is maintained using cross resource status, which checks and wait until the dependencies are provisioned. We will host sets of default templates in this repository. Users can specify which set of templates they intend to use by specifying the corresponding `version` or `channel` in the embedded `CommonSpec` in the component's CR.
146+
147+
Each component's controller will generate necessary certificates for the component and store them to the [secret resources](https://cluster-api.sigs.k8s.io/tasks/certs/using-custom-certificates.html) defined by CAPI. Also, The KAS controller will store the content of the kubeconfig file in a secret named `[clustername]-kubeconfig`.
148+
149+
![Control Plane Creating Process](in-tree.png)
150+
151+
The creating process will include six steps:
152+
153+
1. The user generates all CRs, i.e., NCP, Etcd, APIServer, ControllerManager, with the same namespace, and apply them.
154+
155+
2. The Etcd controller generates the certificates (including a root CA and TLS serving certificates), creates the Etcd workload, and stores the certificates into `secret/[cluster-name]-etcd`
156+
157+
3. The KAS controller creates a KAS service (for exposing the NCP), generates certificates (including a root CA, TLS serving certificates), creates the KAS workload, stores the certificates into `secret/[cluster-name]-ca`, creates a kubeconfig and stores it into `secret/[cluster-name]-kubeconfig`.
158+
159+
4. The KCM controller generates the KCM kubeconfig and creates the KCM workload.
160+
161+
5. After all the three components are ready, the NCP controller marks the NCP CR as ready.
162+
163+
#### Using out-of-tree provisioners
164+
165+
If users intend to use an external controller to create the NCP component, they may need to implement a new component controller that can interact with the component CR and the external controller to create the component. For example, if the user wanted to use the [etcd-cluster-operator](https://github.com/improbable-eng/etcd-cluster-operator) that requires the [EtcdCluster](https://github.com/improbable-eng/etcd-cluster-operator/blob/master/api/v1alpha1/etcdcluster_types.go) CR. They need to implement a custom controller that watches the `NestedEtcd` resource, creates the necessary CRs for that implementation, and updates the required status fields on `NestedEtcd` to allow dependent services to be provisioned. This can be done using the [kubebuilder-declarative-pattern](https://github.com/kubernetes-sigs/kubebuilder-declarative-pattern) like is done for in-tree component controllers.
166+
167+
![Creating a Control Plane using out-of-tree provisioners](out-of-tree.png)
168+
169+
In the following example, we assume that the user intends to use Etcd-cluster-operator(ECO) as the Etcd controller. The creating process will include seven steps:
170+
171+
1. The cluster administrator deletes the in-tree Etcd controller and deploys the custom Etcd controller (ECO controller).
172+
173+
2. The user generates all CRs and apply them.
174+
175+
3. The ECO controller creates the EtcdCluster CR.
176+
177+
4. The ECO creates the Etcd workload.
178+
179+
5. At the meantime, the ECO controller keeps watching the EtcdCluster CR, stores the Etcd CA into the `secret/[cluster-name]-etcd`, and updates the `Etcd` CR accordingly.
180+
181+
6. The KAS controller creates the KAS service, generates certificates, creates the KAS workload, stores certificates into `secret/[cluster-name]-ca`, creates the kubeconfig and stores it into the `secret/[cluster-name]-kubeconfig`
182+
183+
7. The KCM controller generates the KCM kubeconfig and creates the KCM workload.
184+
185+
8. Once all the three components are ready, the NCP controller marks the NCP CR as ready.
186+
187+
### Control Plane Custom Resources
188+
189+
The followings are CRDs of the three components.
190+
191+
### NestedEtcd CRD
192+
```go
193+
// NestedEtcdSpec defines the desired state of Etcd
194+
type NestedEtcdSpec struct {
195+
// NestedComponentSpec contains the common and user-specified information that are
196+
// required for creating the component
197+
// +optional
198+
NestedComponentSpec `json:",inline"`
199+
}
200+
201+
// NestedEtcdStatus defines the observed state of Etcd
202+
type NestedEtcdStatus struct {
203+
// Ready is set if all resources have been created
204+
Ready bool `json:"ready,omitempty"`
205+
206+
// EtcdDomain defines how to address the etcd instance
207+
Addresses []NestedEtcdAddress `json:"addresses,omitempty"`
208+
209+
// CommonStatus allows addons status monitoring
210+
addonv1alpha1. CommonStatus `json:",inline"`
211+
}
212+
213+
// EtcdAddress defines the observed addresses for etcd
214+
type NestedEtcdAddress struct {
215+
// IP Address of the etcd instance.
216+
// +optional
217+
IP string `json:"ip,omitempty"`
218+
219+
// Hostname of the etcd instance
220+
Hostname string `json:"hostname,omitempty"`
221+
222+
// Port of the etcd instance
223+
// +optional
224+
Port int32 `json:"port"`
225+
}
226+
227+
// NestedEtcd is the Schema for the Etcd API
228+
type NestedEtcd struct {
229+
metav1.TypeMeta `json:",inline"`
230+
metav1.ObjectMeta `json:"metadata,omitempty"`
231+
232+
Spec EtcdSpec `json:"spec,omitempty"`
233+
Status EtcdStatus `json:"status,omitempty"`
234+
}
235+
```
236+
237+
### NestedAPIServer CRD
238+
239+
```go
240+
type NestedAPIServerSpec struct {
241+
// NestedComponentSpec contains the common and user-specified information that are
242+
// required for creating the component
243+
// +optional
244+
NestedComponentSpec `json:",inline"`
245+
}
246+
247+
// NestedAPIServerStatus defines the observed state of APIServer
248+
type NestedAPIServerStatus struct {
249+
// Ready is set if all resources have been created
250+
// +kubebuilder:default=false
251+
Ready bool `json:"ready,omitempty"`
252+
253+
// APIServerService is the reference to the service that expose the APIServer
254+
// +optional
255+
APIServerService *corev1.ObjectReference `json:"apiserverService,omitempty"`
256+
257+
// CommonStatus allows addons status monitoring
258+
addonv1alpha1. CommonStatus `json:",inline"`
259+
}
260+
261+
// NestedAPIServer is the Schema for the APIServers API
262+
type NestedAPIServer struct {
263+
metav1.TypeMeta `json:",inline"`
264+
metav1.ObjectMeta `json:"metadata,omitempty"`
265+
266+
Spec NestedAPIServerSpec `json:"spec,omitempty"`
267+
Status NestedAPIServerStatus `json:"status,omitempty"`
268+
}
269+
```
270+
271+
### NestedControllerManager CRD
272+
273+
```go
274+
// NestedControllerManagerSpec defines the desired state of ControllerManager
275+
type NestedControllerManagerSec struct {
276+
// NestedComponentSpec contains the common and user-specified information that are
277+
// required for creating the component
278+
// +optional
279+
NestedComponentSpec `json:",inline"`
280+
}
281+
282+
// NestedControllerManagerStatus defines the observed state of ControllerManager
283+
type NestedControllerManagerStatus struct {
284+
// Ready is set if all resources have been created
285+
Ready bool `json:"ready,omitempty"`
286+
287+
// CommonStatus allows addons status monitoring
288+
addonv1alpha1. CommonStatus `json:",inline"`
289+
}
290+
291+
// NestedControllerManager is the Schema for the ControllerManagers API
292+
type NestedControllerManager struct {
293+
metav1.TypeMeta `json:",inline"`
294+
metav1.ObjectMeta `json:"metadata,omitempty"`
295+
296+
Spec NestedControllerManagerSpec `json:"spec,omitempty"`
297+
Status NestedControllerManagerStatus `json:"status,omitempty"`
298+
}
299+
```
300+
301+
### Security Model
302+
303+
Creating an NCP requires the end-user to submit a creation request, and the
304+
cluster administrator will be responsible for creating the NCP CRs and applying
305+
them. Once the NCP is ready, the cluster administrator will return a kubeconfig
306+
to the end-user, as each end-user can only access the apiserver assigned to them.
307+
There is no need to worry about malicious users to manipulate other users' resources.
308+
A malicious user can still skew the system by creating a massive amount of resources.
309+
To avoid this, we need to enhance the syncer component; however, this topic is
310+
beyond this proposal's scope. The proposed mechanism will not lead to any
311+
severe security issues.
312+
313+
## Implementation History
314+
315+
- [ ] MM/DD/YYYY: Proposed idea in an issue or [community meeting]
316+
- [ ] MM/DD/YYYY: Compile a Google Doc following the CAEP template (link here)
317+
- [ ] MM/DD/YYYY: First round of feedback from community
318+
- [ ] MM/DD/YYYY: Present proposal at a [community meeting]
319+
- [ ] MM/DD/YYYY: Open proposal PR

‎proposals/in-tree.png

34 KB
Loading

‎proposals/out-of-tree.png

46.8 KB
Loading

0 commit comments

Comments
 (0)