Can't use ReadOnlyMany volume restored from Snapshot with controller-publish-readonly provisioner option

[tsmetana]

When using the new controller-publish-readonly command-line option in the external provisioner sidecar, the ReadOnlyMany volumes seem to be still attempted to be mounted read-write. Moreover it looks like when the ROX volume is restored from snapshot, the kubelet still attempts to even create a filesystem on it.

Reproducer:

1. Create a CSI volume:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: myclaim1
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: standard-csi
  resources:
    requests:
      storage: 1Gi

2. Start a pod that writes some data on it (optionally, just to have something for verification).
3. Create a snapshot of the volume:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: mysnap-1
spec:
  volumeSnapshotClassName: csi-gce-pd-vsc
  source:
    persistentVolumeClaimName: myclaim1

4. Restore the snapshot to ReadOnlyMany volume

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc1-restore
spec:
  accessModes:
  - ReadOnlyMany
  dataSource:
    apiGroup: snapshot.storage.k8s.io
    kind: VolumeSnapshot
    name: mysnap-1
  resources:
    requests:
      storage: 1Gi
  storageClassName: standard-csi

5. Try to use the restored volume in a pod:

kind: Pod
apiVersion: v1
metadata:
  name: pod-restore
  labels:
    name: pod-restore
spec:
  containers:
    - name: busybox
      image: gcr.io/google_containers/busybox
      command:
        - "/bin/sh"
        - "-c"
        - "while true; do sleep 1; done"
      volumeMounts:
      - mountPath: "/mnt/test"
        name: vol
        readOnly: true
  volumes:
    - name: vol
      persistentVolumeClaim:
        claimName: pvc1-restore
        readOnly: true

Result:
Pod fails to start with message similar to this one in the log:

Warning  FailedMount             16s (x7 over 51s)  kubelet                  MountVolume.MountDevice failed for volume "pvc-e4269d3a-3a2d-4920-8917-a30c0a0773e7" : rpc error: code = Internal desc = Failed to format and mount device from ("/dev/disk/by-id/google-pvc-e4269d3a-3a2d-4920-8917-a30c0a0773e7") to ("/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e4269d3a-3a2d-4920-8917-a30c0a0773e7/globalmount") with fstype ("ext4") and options ([]): format of disk "/dev/disk/by-id/google-pvc-e4269d3a-3a2d-4920-8917-a30c0a0773e7" failed: type:("ext4") target:("/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e4269d3a-3a2d-4920-8917-a30c0a0773e7/globalmount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
/dev/disk/by-id/google-pvc-e4269d3a-3a2d-4920-8917-a30c0a0773e7: Read-only file system while setting up superblock

This looks to be incorrect: the volume should not be attempted to be mounted read-write and the attempt to create a filesystem on a formatted drive pre-populated with data looks quite dangerous. I will try to tinker with this but if someone reading this had an idea, I'm interested. I'm also not sure who exactly is at fault here: might be also kubelet.

[tsmetana]

cc: @humblec -- might be interesting to you...

[mattcary]

/cc @leiyiz

Leiyi has just disabled rw mounting for rox volumes, or at least tried to.

It looks like the disk is actually mounted read-only, maybe there was some problem with the filesystem that made the driver think it wasn't formatted correctly?

[leiyiz]

will need to try that out to be sure... yeah I think mounter somehow think there's no filesystem?

[humblec]

@tsmetana can you please list the PV spec associated with pvc1-restore PVC ? also just wondering what are the capabilities (like controller publish, node publish..etc) this this CSI driver (gce) support.

[mattcary]

@humblec

https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/blob/master/pkg/gce-pd-csi-driver/gce-pd-driver.go#L63

controller publish (GCE PD attach to VM), node stage (mount on host), node publish (bind mount to host mount)

[leiyiz]

the fix is out and this workflow should work once the fix is merged and released

[humblec]

@humblec

https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/blob/master/pkg/gce-pd-csi-driver/gce-pd-driver.go#L63

controller publish (GCE PD attach to VM), node stage (mount on host), node publish (bind mount to host mount)

Thanks @mattcary , one other thing I wanted to make sure here was the readonly presence in the PV CSI source once the flag (--controller-publish) is ON and the PVC carried ROX access mode. As I dont have a GCP setup i was looking for describe output of a PV.

[mattcary]

Here's a PV/PVC pair from a GKE cluster (before @leiyiz's fix). Note the PV is ROX but readOnly is not set in the csi spec.

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: pd.csi.storage.gke.io
  creationTimestamp: "2021-12-08T16:06:47Z"
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/pd-csi-storage-gke-io
  name: pvc-384684ef-e1e0-4d8f-ac66-b1fc90417669
  resourceVersion: "29093513"
  uid: 476d1030-4ccc-4c27-8fe3-6b6a51a62d5f
spec:
  accessModes:
  - ReadOnlyMany
  capacity:
    storage: 10Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: busy-pvc
    namespace: default
    resourceVersion: "29093475"
    uid: 384684ef-e1e0-4d8f-ac66-b1fc90417669
  csi:
    driver: pd.csi.storage.gke.io
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: 1636001418599-8081-pd.csi.storage.gke.io
    volumeHandle: projects/mattcary-gke-dev2/zones/us-central1-b/disks/pvc-384684ef-e1e0-4d8f-ac66-b1fc90417669
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.gke.io/zone
          operator: In
          values:
          - us-central1-b
  persistentVolumeReclaimPolicy: Delete
  storageClassName: standard-rwo
  volumeMode: Block
status:
  phase: Bound

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
    volume.kubernetes.io/selected-node: gke-default-default-pool-f208f925-ecaw
  creationTimestamp: "2021-12-08T16:06:42Z"
  finalizers:
  - kubernetes.io/pvc-protection
  name: busy-pvc
  namespace: default
  resourceVersion: "29093501"
  uid: 384684ef-e1e0-4d8f-ac66-b1fc90417669
spec:
  accessModes:
  - ReadOnlyMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: standard-rwo
  volumeMode: Block
  volumeName: pvc-384684ef-e1e0-4d8f-ac66-b1fc90417669
status:
  accessModes:
  - ReadOnlyMany
  capacity:
    storage: 10Gi
  phase: Bound

[leiyiz]

one other thing I wanted to make sure here was the readonly presence in the PV CSI source once the flag (--controller-publish) is ON and the PVC carried ROX access mode.

if you are using external provisioner 3.0.0 and that flag is set, PV should have the csi.readonly set. If that is what you are asking