Merge pull request #136 from msau42/topology

add topology docs

Author: k8s-ci-robot

book/src/SUMMARY.md

@@ -11,7 +11,7 @@
         - [livenessprobe](livenessprobe.md)
     - [CSI objects](csi-objects.md)
       - [CSIDriver Object](csi-driver-object.md)
-      - [CSINodeInfo Object](csi-node-info-object.md)
+      - [CSINode Object](csi-node-object.md)
     - [Features](features.md)
         - [Secrets & Credentials](secrets-and-credentials.md)
         - [Volume Snapshot & Restore](snapshot-restore-feature.md)

book/src/csi-node-info-object.md renamed to book/src/csi-node-object.md

@@ -1,69 +1,65 @@
-# CSINodeInfo Object
+# CSINode Object
 
 ## Status
 
 * Kubernetes 1.12 - 1.13: Alpha
 * Kubernetes 1.14: Beta
 
-## What is the CSINodeInfo object?
+## What is the CSINode object?
 
-CSI drivers generate node specific information. Instead of storing this in the Kubernetes `Node` API Object, a new CSI specific Kubernetes `CSINodeInfo` object was created.
+CSI drivers generate node specific information. Instead of storing this in the Kubernetes `Node` API Object, a new CSI specific Kubernetes `CSINode` object was created.
 
 It serves the following purposes:
 
 1. Mapping Kubernetes node name to CSI Node name,
-  * The CSI `GetNodeInfo` call returns the name by which the storage system refers to a node. Kubernetes must use this name in future `ControllerPublishVolume` calls. Therefore, when a new CSI driver is registered, Kubernetes stores the storage system node ID in the `CSINodeInfo` object for future reference. 
+  * The CSI `GetNodeInfo` call returns the name by which the storage system refers to a node. Kubernetes must use this name in future `ControllerPublishVolume` calls. Therefore, when a new CSI driver is registered, Kubernetes stores the storage system node ID in the `CSINode` object for future reference.
 2. Driver availability
   * A way for kubelet to communicate to the kube-controller-manager and kubernetes scheduler whether the driver is available (registered) on the node or not.
 3. Volume topology
-  * The CSI `GetNodeInfo` call returns a set of keys/values labels identifying the topology of that node. Kubernetes uses this information to to do topology-aware provisioning (see [PVC Volume Binding Modes](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode) for more details). It stores the key/values as labels on the Kubernetes node object. In order to recall which `Node` label keys belong to a specific CSI driver, the kubelet stores the keys in the `CSINodeInfo` object for future reference.
-	
+  * The CSI `GetNodeInfo` call returns a set of keys/values labels identifying the topology of that node. Kubernetes uses this information to to do topology-aware provisioning (see [PVC Volume Binding Modes](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode) for more details). It stores the key/values as labels on the Kubernetes node object. In order to recall which `Node` label keys belong to a specific CSI driver, the kubelet stores the keys in the `CSINode` object for future reference.
 
-## What fields does the CSINodeInfo object have?
+## What fields does the CSINode object have?
 
-Here is an example of a v1alpha1 `CSINodeInfo` object:
+Here is an example of a v1beta1 `CSINode` object:
 
 ```YAML
 apiVersion: storage.k8s.io/v1beta1
 kind: CSINodeInfo
 metadata:
   name: node1
 spec:
-  drivers:
-  - name: mycsidriver.example.com
-    available: true
-    volumePluginMechanism: csi-plugin
-status:
   drivers:
   - name: mycsidriver.example.com
     nodeID: storageNodeID1
     topologyKeys: ['mycsidriver.example.com/regions', "mycsidriver.example.com/zones"]
 ```
 
-Where the fields mean:
+What the fields mean:
 
-- `csiDrivers` - list of CSI drivers running on the node and their properties.
-- `driver` - the CSI driver that this object refers to.
+- `drivers` - list of CSI drivers running on the node and their properties.
+- `name` - the CSI driver that this object refers to.
 - `nodeID` - the assigned identifier for the node as determined by the driver.
 - `topologyKeys` - A list of topology keys assigned to the node as supported by the driver.
 
-## What creates the CSINodeInfo object?
+## What creates the CSINode object?
 
-CSI drivers do not need to create the `CSINodeInfo` object directly. Instead they should use the [node-driver-registrar](node-driver-registrar.md) sidecar container. This sidecar container will interact with kubelet via the kubelet plugin registration mechanism to automatically populate the `CSINodeInfo` object on behalf of the the CSI driver.
+CSI drivers do not need to create the `CSINode` object directly. Instead they should use the [node-driver-registrar](node-driver-registrar.md) sidecar container. This sidecar container will interact with kubelet via the kubelet plugin registration mechanism to automatically populate the `CSINode` object on behalf of the the CSI driver.
 
 ## Changes from Alpha to Beta
+
 ### CRD to Built in Type
-During alpha development, the `CSINodeInfo` object was also defined as a [Custom Resource Definition](https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#create-a-customresourcedefinition) (CRD). As part of the promotion to beta the object has been moved to the built-in Kubernetes API.
+The alpha object was called `CSINodeInfo`, whereas the beta object is called
+CSINode`. The alpha `CSINodeInfo` object was also defined as a [Custom Resource Definition](https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#create-a-customresourcedefinition) (CRD). As part of the promotion to beta the object has been moved to the built-in Kubernetes API.
 
 In the move from alpha to beta, the API Group for this object changed from `csi.storage.k8s.io/v1alpha1` to `storage.k8s.io/v1beta1`.
 
 There is no automatic update of existing CRDs and their CRs during Kubernetes update to the new build-in type.
 
 ### Enabling CSINodeInfo on Kubernetes
-In Kubernetes v1.12 and v1.13, because the feature was alpha, it was disabled by default. To enable the use of `CSIDriver` on these versions, do the following:
+In Kubernetes v1.12 and v1.13, because the feature was alpha, it was disabled by default. To enable the use of `CSINodeInfo` on these versions, do the following:
 
 1. Ensure the feature gate is enabled with `--feature-gates=CSINodeInfo=true`
-2. Either ensure the `CSIDriver` CRD is automatically installed via the [Kubernetes Storage CRD addon](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/storage-crds) OR manually install the `CSINodeInfo` CRD on the Kubernetes cluster with the following command:
+2. Either ensure the `CSIDriver` CRD is automatically installed via the [Kubernetes Storage CRD addon](https://github.com/kubernetes/kubernetes/tree/release-1.13/cluster/addons/storage-crds) OR manually install the `CSINodeInfo` CRD on the Kubernetes cluster with the following command:
 
 ```
 $> kubectl create -f https://raw.githubusercontent.com/kubernetes/csi-api/master/pkg/crd/manifests/csinodeinfo.yaml

book/src/csi-objects.md

@@ -5,7 +5,7 @@
 The Kubernetes API contains the following CSI specific objects:
 
 * [CSIDriver Object](csi-driver-object.md)
-* [CSINodeInfo Object](csi-node-info-object.md)
+* [CSINode Object](csi-node-object.md)
 
 Both are part of `storage.k8s.io/v1beta1` API group.
 

book/src/topology.md

@@ -1,8 +1,8 @@
 # CSI Topology Feature
 
-> ## *This page is still under active development.*
-
-**Status:** Alpha
+## Status
+* Kubernetes 1.12: Alpha
+* Kubernetes 1.14: Beta
 
 Some storage systems expose volumes that are not equally accessible by all nodes in a Kubernetes cluster. Instead volumes may be constrained to some subset of node(s) in the cluster. The cluster may be segmented into, for example, “racks” or “regions” and “zones” or some other grouping, and a given volume may be accessible only from one of those groups.
 
@@ -16,24 +16,52 @@ Kubernetes and the Kubernetes CSI [Sidecar Containers](sidecar-containers.md) us
 
 ## Implementing Topology
 
-TODO: Explain the CSI calls and capabilities that must be implemented.
-TODO: Explain what CSI CRDs the feature depends on.
+To support topology in a CSI driver, the following must be implemented:
+
+* The `PluginCapability` must support `VOLUME_ACCESSIBILITY_CONTRAINTS`.
+* The plugin must fill in `accessible_topology` in `NodeGetInfoResponse`.
+  This information will be used to populate the Kubernetes [CSINode object](csi-node-object.md) and add the topology labels to the Node object.
+* During `CreateVolume`, the topology information will get passed in through `CreateVolumeRequest.accessibility_requirements`.
+
+In the StorageClass object, both `volumeBindingMode` values of `Immediate` and
+`WaitForFirstConsumer` are supported.
+
+* If `Immediate` is set, then the
+  external-provisioner will pass in all available topologies in the cluster for
+  the driver.
+* If `WaitForFirstConsumer` is set, then the external-provisioner will wait for
+  the scheduler to pick a node. The topology of that selected node will then be
+  set as the first entry in `CreateVolumeRequest.accessibility_requirements.preferred`.
+  All remaining topologies are still included in the `requisite` and `preferred`
+  fields to support storage systems that span across multiple topologies.
 
-## Usage
+## Beta Usage
 
 In order to support topology-aware dynamic provisioning mechanisms available in Kubernetes, the *external-provisioner* must have the Topology feature enabled:
 
 ```
 --feature-gates=Topology=true
 ```
 
-In addition, in the *Kubernetes cluster* the `CSINodeInfo` alpha feature must be enabled (refer to the [CSINodeInfo Object](csi-node-info-object.md) section for more info):
+In addition, in the *Kubernetes cluster* the `CSINodeInfo` feature must be enabled on both Kubernetes master and nodes (refer to the [CSINode Object](csi-node-object.md) section for more info):
 
 ```
 --feature-gates=CSINodeInfo=true
 ```
 
-The `KubeletPluginsWatcher` feature must also be enabled (GA and enabled by default in Kubernetes 1.13).
+In order to fully function properly, all Kubernetes master and nodes must be on at least
+Kubernetes 1.14. If a selected node is on a lower version, topology is ignored and not
+passed to the driver during `CreateVolume`.
+
+## Alpha Usage
+
+The Kubernetes 1.13 alpha feature requires external-provisioner v1.0.1, and the
+1.12 alpha feature requires external-provisioner v0.4.1. Kubernetes master and
+node version skew and upgrades are not supported.
+
+The `Topology` feature gate must be enabled on the external-provisioner. The
+`CSINodeInfo`, `VolumeScheduling`, and `KubeletPluginsWatcher` feature gates
+must be enabled on both Kubernetes master and nodes.
 
 ## Storage Internal Topology