diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md index 84ec9087..59f0099b 100644 --- a/book/src/SUMMARY.md +++ b/book/src/SUMMARY.md @@ -18,6 +18,7 @@ - [cluster-driver-registrar](cluster-driver-registrar.md) (deprecated) - [external-health-monitor-controller](external-health-monitor-controller.md) - [external-health-monitor-agent](external-health-monitor-agent.md) + - [external-snapshot-metadata](external-snapshot-metadata.md) - [CSI objects](csi-objects.md) - [CSIDriver Object](csi-driver-object.md) - [CSINode Object](csi-node-object.md) @@ -43,6 +44,7 @@ - [CSI Windows](csi-windows.md) - [Volume Mode Conversion](prevent-volume-mode-conversion.md) - [Cross-Namespace Data Sources](cross-namespace-data-sources.md) + - [Changed Block Tracking](changed-block-tracking.md) - [Deploying a CSI Driver on Kubernetes](deploying.md) - [Example](example.md) - [Driver Testing](testing-drivers.md) diff --git a/book/src/changed-block-tracking.md b/book/src/changed-block-tracking.md new file mode 100644 index 00000000..59dab34e --- /dev/null +++ b/book/src/changed-block-tracking.md @@ -0,0 +1,37 @@ +# Changed Block Tracking + +## Status + +Status | Min K8s Version | Max K8s Version | Min CSI Version | Max CSI Version | +-------|-----------------|-----------------|-----------------|-----------------| +Alpha | 1.33 | - | [v1.10.0](https://github.com/container-storage-interface/spec/releases/tag/v1.10.0) | - + + +## Overview + +This optional feature provides a secure mechanism to obtain metadata +on the allocated blocks of a CSI VolumeSnapshot, or the changed blocks between two arbitrary pairs of CSI VolumeSnapshot objects of the same PersistentVolume. +See [Kubernetes Enhancement Proposal 3314](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3314-csi-changed-block-tracking) +for full details. + +A CSI driver must advertise its support for this feature by specifying the +`SNAPSHOT_METADATA_SERVICE` capability in the response of the +[GetPluginCapabilities](https://github.com/container-storage-interface/spec/blob/master/spec.md#getplugincapabilities) +RPC (part of the +[CSI Identity Service](https://github.com/container-storage-interface/spec/blob/master/spec.md#identity-service-rpc)), +and must implement the +[CSI SnapshotMetadata Service](https://github.com/container-storage-interface/spec/blob/master/spec.md#snapshot-metadata-service-rpcs). + +CSI drivers that implement this feature should deploy the +[external-snapshot-metadata](./external-snapshot-metadata.md) +sidecar and create the associated +[Snapshot Metadata Service CR](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3314-csi-changed-block-tracking#snapshot-metadata-service-custom-resource). + +The `external-snapshot-metadata` sidecar implements the server side of the +[Kubernetes SnapshotMetadata Service API](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3314-csi-changed-block-tracking#the-kubernetes-snapshotmetadata-service-api). +Backup applications obtain VolumeSnapshot metadata directly from the sidecar +through this API, bypassing the Kubernetes API server for the most part. +Backup application developers should refer to the [Usage](external-snapshot-metadata.md#usage) +and [Resources](external-snapshot-metadata.md#resources) +sections of the sidecar documentation for details. + diff --git a/book/src/example.md b/book/src/example.md index 5c387630..f6725360 100644 --- a/book/src/example.md +++ b/book/src/example.md @@ -5,4 +5,13 @@ The [deployment example](https://github.com/kubernetes-csi/csi-driver-host-path# The example deployment uses the original RBAC rule files that are maintained together with sidecar apps and deploys into the default namespace. A real production should copy the RBAC files and customize them as explained in the comments of those files. +The Hostpath CSI driver only supports the [Changed Block Tracking](./changed-block-tracking.md) feature for +PersistentVolumes with `Block` VolumeMode. +The support is not enabled by default, but requires additional configuration when deploying the driver. +When enabled, the +[external-snapshot-metadata](./external-snapshot-metadata.md) +sidecar will also be deployed, and the additional RBAC policies needed will get created. +See the [docs/example-snapshot-metadata.md](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/docs/example-snapshot-metadata.md) +document for details. + If you encounter any problems, please check the [Troubleshooting page](troubleshooting.html). diff --git a/book/src/external-snapshot-metadata.md b/book/src/external-snapshot-metadata.md new file mode 100644 index 00000000..cefbfe2f --- /dev/null +++ b/book/src/external-snapshot-metadata.md @@ -0,0 +1,118 @@ +# CSI external-snapshot-metadata + +## Status and Releases + +**Git Repository:** [https://github.com/kubernetes-csi/external-snapshot-metadata](https://github.com/kubernetes-csi/external-snapshot-metadata) + +### Supported Versions + +Latest stable release | Branch | Min CSI Version | Max CSI Version | Container Image | [Min K8s Version](project-policies.md#minimum-version) | [Max K8s Version](project-policies.md#maximum-version) | [Recommended K8s Version](project-policies.md#recommended-version) | +--|--|--|--|--|--|--|-- +v0.1.0 | [v0.1.0](https://github.com/kubernetes-csi/external-snapshot-metadata/releases/tag/v0.1.0) | [v1.10.0](https://github.com/container-storage-interface/spec/releases/tag/v1.10.0) | - | registry.k8s.io/sig-storage/csi-snapshot-metadata:v0.1.0 | v1.33 | - | v1.33 + + +## Alpha + +### Description +This sidecar securely serves snapshot metadata to Kubernetes clients through the +[Kubernetes SnapshotMetadata Service API](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3314-csi-changed-block-tracking#the-kubernetes-snapshotmetadata-service-api). +This API is similar to the +[CSI SnapshotMetadata Service](https://github.com/container-storage-interface/spec/blob/master/spec.md#snapshot-metadata-service-rpcs) +but is designed to be used by Kuberetes authenticated and authorized backup applications. +Its [protobuf specification](https://github.com/kubernetes-csi/external-snapshot-metadata/tree/main/proto/schema.proto) +is available in the sidecar repository. + +The sidecar authenticates and authorizes each Kubernetes backup application request made through the +Kubernetes SnapshotMetadata Service API. +It then acts as a proxy as it fetches the desired metadata from the CSI driver and +streams it directly to the requesting application with no load on the Kubernetes API server. + +See ["The External Snapshot Metadata Sidecar"](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3314-csi-changed-block-tracking#the-external-snapshot-metadata-sidecar) +section in the CSI Changed Block Tracking KEP for additional details on the sidecar. + +### Usage +Backup applications, identified by their authorized ServiceAccount objects, +directly communicate with the sidecar using the +[Kubernetes SnapshotMetadata Service API](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3314-csi-changed-block-tracking#the-kubernetes-snapshotmetadata-service-api). +The authorization needed is described in the +["Risks and Mitigations"](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3314-csi-changed-block-tracking#risks-and-mitigations) +section of the CSI Changed Block Tracking KEP. +In particular, this requires the ability to use the Kubernetes +[TokenRequest](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-request-v1/) +API and to access the objects required to use the API. + +The availability of this optional feature is advertised to backup applications by the presence of a +[Snapshot Metadata Service CR](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3314-csi-changed-block-tracking#snapshot-metadata-service-custom-resource) +that is named for the CSI driver that provisioned the PersistentVolume and VolumeSnapshot objects involved. +The CR contains a gRPC service endpoint and CA certificate, and an audience string for authentication. +A backup application must use the Kubernetes +[TokenRequest](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-request-v1/) +API with the audience string to obtain a Kubernetes authentication token for use in the +Kubernetes SnapshotMetadata Service API call. +The backup application should establish trust for the CA certificate before making the gRPC call +to the service endpoint. + +The Kubernetes SnapshotMetadata Service API uses a gRPC stream to return VolumeSnapshot metadata +to the backup application. Metadata can be lengthy, so the API supports +restarting an interrupted metadata request from an intermediate point in case of failure. +The [Resources](external-snapshot-metadata.md#resources) section below describes the +programming artifacts available to support backup application use of this API. + +### Deployment +The CSI `external-snapshot-metadata` sidecar should be deployed by +CSI drivers that support the +[Changed Block Tracking](./changed-block-tracking.md) feature. +The sidecar must be deployed in the same pod as the CSI driver and +will communicate with its gRPC [CSI SnapshotMetadata Service](https://github.com/container-storage-interface/spec/blob/master/spec.md#snapshot-metadata-service-rpcs) +and [CSI Identity Service](https://github.com/container-storage-interface/spec/blob/master/spec.md#identity-service-rpc) +over a UNIX domain socket. + +The sidecar should be configured to run under the authority of its +CSI driver ServiceAccount, which must be authorized as described in the +["Risks and Mitigations"](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3314-csi-changed-block-tracking#risks-and-mitigations) +section of the CSI Changed Block Tracking KEP. +In particular, this requires the ability to use the Kubernetes +[TokenReview](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-review-v1/) +and +[SubjectAccessReview](https://kubernetes.io/docs/reference/kubernetes-api/authorization-resources/subject-access-review-v1/) +APIs. + +A Service object must be created to expose the endpoints of the +[Kubernetes SnapshotMetadata Service API](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3314-csi-changed-block-tracking#the-kubernetes-snapshotmetadata-service-api) +gRPC server implemented by the sidecar. + +A [SnapshotMetadataService CR](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3314-csi-changed-block-tracking#snapshot-metadata-service-custom-resource), +named for the CSI driver, must be created to advertise the +availability of this optional feature to Kubernetes backup application clients. +The CR contains the CA certificate and Service endpoint address +of the sidecar and the audience string needed for the client +authentication token. + +### Resources + +The [external-snapshot-metadata repository](https://github.com/kubernetes-csi/external-snapshot-metadata) contains +the [protobuf specification](https://github.com/kubernetes-csi/external-snapshot-metadata/tree/main/proto/schema.proto) +of the +[Kubernetes SnapshotMetadata Service API](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3314-csi-changed-block-tracking#the-kubernetes-snapshotmetadata-service-api). + +In addition, the repository has a number of useful artifacts to support Go language programs: + +- The +[Go client interface](https://github.com/kubernetes-csi/external-snapshot-metadata/tree/main/client) +for the Kubernetes SnapshotMetadata Service API. + +- The +[pkg/iterator](https://github.com/kubernetes-csi/external-snapshot-metadata/tree/main/pkg/iterator) +client helper package. +This may be used by backup applications instead of working directly with the +Kubernetes SnapshotMetadata Service gRPC client interfaces. +The +[snapshot-metadata-lister](https://github.com/kubernetes-csi/external-snapshot-metadata/tree/main/examples/snapshot-metadata-lister) +example Go command illustrates the use of this helper package. + +- Go language +[CSI SnapshotMetadataService API client mocks](https://github.com/kubernetes-csi/external-snapshot-metadata/tree/main/pkg/csiclientmocks). + +The sample [Hostpath CSI driver](example.md) has been extended to support the +[Changed Block Tracking](./changed-block-tracking.md) feature +and provides an illustration on how to deploy a CSI driver.