Skip to content

Commit 2849e1f

Browse files
authored
Merge pull request #136 from msau42/topology
add topology docs
2 parents 110683a + 2fb0ac8 commit 2849e1f

File tree

4 files changed

+55
-31
lines changed

4 files changed

+55
-31
lines changed

book/src/SUMMARY.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
- [livenessprobe](livenessprobe.md)
1212
- [CSI objects](csi-objects.md)
1313
- [CSIDriver Object](csi-driver-object.md)
14-
- [CSINodeInfo Object](csi-node-info-object.md)
14+
- [CSINode Object](csi-node-object.md)
1515
- [Features](features.md)
1616
- [Secrets & Credentials](secrets-and-credentials.md)
1717
- [Volume Snapshot & Restore](snapshot-restore-feature.md)

book/src/csi-node-info-object.md renamed to book/src/csi-node-object.md

Lines changed: 17 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,69 +1,65 @@
1-
# CSINodeInfo Object
1+
# CSINode Object
22

33
## Status
44

55
* Kubernetes 1.12 - 1.13: Alpha
66
* Kubernetes 1.14: Beta
77

8-
## What is the CSINodeInfo object?
8+
## What is the CSINode object?
99

10-
CSI drivers generate node specific information. Instead of storing this in the Kubernetes `Node` API Object, a new CSI specific Kubernetes `CSINodeInfo` object was created.
10+
CSI drivers generate node specific information. Instead of storing this in the Kubernetes `Node` API Object, a new CSI specific Kubernetes `CSINode` object was created.
1111

1212
It serves the following purposes:
1313

1414
1. Mapping Kubernetes node name to CSI Node name,
15-
* The CSI `GetNodeInfo` call returns the name by which the storage system refers to a node. Kubernetes must use this name in future `ControllerPublishVolume` calls. Therefore, when a new CSI driver is registered, Kubernetes stores the storage system node ID in the `CSINodeInfo` object for future reference.
15+
* The CSI `GetNodeInfo` call returns the name by which the storage system refers to a node. Kubernetes must use this name in future `ControllerPublishVolume` calls. Therefore, when a new CSI driver is registered, Kubernetes stores the storage system node ID in the `CSINode` object for future reference.
1616
2. Driver availability
1717
* A way for kubelet to communicate to the kube-controller-manager and kubernetes scheduler whether the driver is available (registered) on the node or not.
1818
3. Volume topology
19-
* The CSI `GetNodeInfo` call returns a set of keys/values labels identifying the topology of that node. Kubernetes uses this information to to do topology-aware provisioning (see [PVC Volume Binding Modes](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode) for more details). It stores the key/values as labels on the Kubernetes node object. In order to recall which `Node` label keys belong to a specific CSI driver, the kubelet stores the keys in the `CSINodeInfo` object for future reference.
20-
19+
* The CSI `GetNodeInfo` call returns a set of keys/values labels identifying the topology of that node. Kubernetes uses this information to to do topology-aware provisioning (see [PVC Volume Binding Modes](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode) for more details). It stores the key/values as labels on the Kubernetes node object. In order to recall which `Node` label keys belong to a specific CSI driver, the kubelet stores the keys in the `CSINode` object for future reference.
2120

22-
## What fields does the CSINodeInfo object have?
21+
## What fields does the CSINode object have?
2322

24-
Here is an example of a v1alpha1 `CSINodeInfo` object:
23+
Here is an example of a v1beta1 `CSINode` object:
2524

2625
```YAML
2726
apiVersion: storage.k8s.io/v1beta1
2827
kind: CSINodeInfo
2928
metadata:
3029
name: node1
3130
spec:
32-
drivers:
33-
- name: mycsidriver.example.com
34-
available: true
35-
volumePluginMechanism: csi-plugin
36-
status:
3731
drivers:
3832
- name: mycsidriver.example.com
3933
nodeID: storageNodeID1
4034
topologyKeys: ['mycsidriver.example.com/regions', "mycsidriver.example.com/zones"]
4135
```
4236
43-
Where the fields mean:
37+
What the fields mean:
4438
45-
- `csiDrivers` - list of CSI drivers running on the node and their properties.
46-
- `driver` - the CSI driver that this object refers to.
39+
- `drivers` - list of CSI drivers running on the node and their properties.
40+
- `name` - the CSI driver that this object refers to.
4741
- `nodeID` - the assigned identifier for the node as determined by the driver.
4842
- `topologyKeys` - A list of topology keys assigned to the node as supported by the driver.
4943

50-
## What creates the CSINodeInfo object?
44+
## What creates the CSINode object?
5145

52-
CSI drivers do not need to create the `CSINodeInfo` object directly. Instead they should use the [node-driver-registrar](node-driver-registrar.md) sidecar container. This sidecar container will interact with kubelet via the kubelet plugin registration mechanism to automatically populate the `CSINodeInfo` object on behalf of the the CSI driver.
46+
CSI drivers do not need to create the `CSINode` object directly. Instead they should use the [node-driver-registrar](node-driver-registrar.md) sidecar container. This sidecar container will interact with kubelet via the kubelet plugin registration mechanism to automatically populate the `CSINode` object on behalf of the the CSI driver.
5347

5448
## Changes from Alpha to Beta
49+
5550
### CRD to Built in Type
56-
During alpha development, the `CSINodeInfo` object was also defined as a [Custom Resource Definition](https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#create-a-customresourcedefinition) (CRD). As part of the promotion to beta the object has been moved to the built-in Kubernetes API.
51+
The alpha object was called `CSINodeInfo`, whereas the beta object is called
52+
CSINode`. The alpha `CSINodeInfo` object was also defined as a [Custom Resource Definition](https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#create-a-customresourcedefinition) (CRD). As part of the promotion to beta the object has been moved to the built-in Kubernetes API.
5753

5854
In the move from alpha to beta, the API Group for this object changed from `csi.storage.k8s.io/v1alpha1` to `storage.k8s.io/v1beta1`.
5955

6056
There is no automatic update of existing CRDs and their CRs during Kubernetes update to the new build-in type.
6157

6258
### Enabling CSINodeInfo on Kubernetes
63-
In Kubernetes v1.12 and v1.13, because the feature was alpha, it was disabled by default. To enable the use of `CSIDriver` on these versions, do the following:
59+
In Kubernetes v1.12 and v1.13, because the feature was alpha, it was disabled by default. To enable the use of `CSINodeInfo` on these versions, do the following:
6460

6561
1. Ensure the feature gate is enabled with `--feature-gates=CSINodeInfo=true`
66-
2. Either ensure the `CSIDriver` CRD is automatically installed via the [Kubernetes Storage CRD addon](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/storage-crds) OR manually install the `CSINodeInfo` CRD on the Kubernetes cluster with the following command:
62+
2. Either ensure the `CSIDriver` CRD is automatically installed via the [Kubernetes Storage CRD addon](https://github.com/kubernetes/kubernetes/tree/release-1.13/cluster/addons/storage-crds) OR manually install the `CSINodeInfo` CRD on the Kubernetes cluster with the following command:
6763

6864
```
6965
$> kubectl create -f https://raw.githubusercontent.com/kubernetes/csi-api/master/pkg/crd/manifests/csinodeinfo.yaml

book/src/csi-objects.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
The Kubernetes API contains the following CSI specific objects:
66

77
* [CSIDriver Object](csi-driver-object.md)
8-
* [CSINodeInfo Object](csi-node-info-object.md)
8+
* [CSINode Object](csi-node-object.md)
99

1010
Both are part of `storage.k8s.io/v1beta1` API group.
1111

book/src/topology.md

Lines changed: 36 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
# CSI Topology Feature
22

3-
> ## *This page is still under active development.*
4-
5-
**Status:** Alpha
3+
## Status
4+
* Kubernetes 1.12: Alpha
5+
* Kubernetes 1.14: Beta
66

77
Some storage systems expose volumes that are not equally accessible by all nodes in a Kubernetes cluster. Instead volumes may be constrained to some subset of node(s) in the cluster. The cluster may be segmented into, for example, “racks” or “regions” and “zones” or some other grouping, and a given volume may be accessible only from one of those groups.
88

@@ -16,24 +16,52 @@ Kubernetes and the Kubernetes CSI [Sidecar Containers](sidecar-containers.md) us
1616

1717
## Implementing Topology
1818

19-
TODO: Explain the CSI calls and capabilities that must be implemented.
20-
TODO: Explain what CSI CRDs the feature depends on.
19+
To support topology in a CSI driver, the following must be implemented:
20+
21+
* The `PluginCapability` must support `VOLUME_ACCESSIBILITY_CONTRAINTS`.
22+
* The plugin must fill in `accessible_topology` in `NodeGetInfoResponse`.
23+
This information will be used to populate the Kubernetes [CSINode object](csi-node-object.md) and add the topology labels to the Node object.
24+
* During `CreateVolume`, the topology information will get passed in through `CreateVolumeRequest.accessibility_requirements`.
25+
26+
In the StorageClass object, both `volumeBindingMode` values of `Immediate` and
27+
`WaitForFirstConsumer` are supported.
28+
29+
* If `Immediate` is set, then the
30+
external-provisioner will pass in all available topologies in the cluster for
31+
the driver.
32+
* If `WaitForFirstConsumer` is set, then the external-provisioner will wait for
33+
the scheduler to pick a node. The topology of that selected node will then be
34+
set as the first entry in `CreateVolumeRequest.accessibility_requirements.preferred`.
35+
All remaining topologies are still included in the `requisite` and `preferred`
36+
fields to support storage systems that span across multiple topologies.
2137

22-
## Usage
38+
## Beta Usage
2339

2440
In order to support topology-aware dynamic provisioning mechanisms available in Kubernetes, the *external-provisioner* must have the Topology feature enabled:
2541

2642
```
2743
--feature-gates=Topology=true
2844
```
2945

30-
In addition, in the *Kubernetes cluster* the `CSINodeInfo` alpha feature must be enabled (refer to the [CSINodeInfo Object](csi-node-info-object.md) section for more info):
46+
In addition, in the *Kubernetes cluster* the `CSINodeInfo` feature must be enabled on both Kubernetes master and nodes (refer to the [CSINode Object](csi-node-object.md) section for more info):
3147

3248
```
3349
--feature-gates=CSINodeInfo=true
3450
```
3551

36-
The `KubeletPluginsWatcher` feature must also be enabled (GA and enabled by default in Kubernetes 1.13).
52+
In order to fully function properly, all Kubernetes master and nodes must be on at least
53+
Kubernetes 1.14. If a selected node is on a lower version, topology is ignored and not
54+
passed to the driver during `CreateVolume`.
55+
56+
## Alpha Usage
57+
58+
The Kubernetes 1.13 alpha feature requires external-provisioner v1.0.1, and the
59+
1.12 alpha feature requires external-provisioner v0.4.1. Kubernetes master and
60+
node version skew and upgrades are not supported.
61+
62+
The `Topology` feature gate must be enabled on the external-provisioner. The
63+
`CSINodeInfo`, `VolumeScheduling`, and `KubeletPluginsWatcher` feature gates
64+
must be enabled on both Kubernetes master and nodes.
3765

3866
## Storage Internal Topology
3967

0 commit comments

Comments
 (0)