Deployment of driver fails on GKE: Unknown user

[saad-ali]

I see the following when deploying to GKE:

2018-07-16 07:35:05.000 PDT
Unknown user "system:serviceaccount:default:csi-controller-sa"

But kubectl shows that the account exists:

$ kubectl get sa
NAME                SECRETS   AGE
csi-controller-sa   1         8m
csi-node-sa         1         8m
default             1         1h

Also see this error:

E  github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:serviceaccount:default:csi-controller-sa" cannot list persistentvolumeclaims at the cluster scope: [clusterrole.rbac.authorization.k8s.io "system:csi-external-attacher" not found, clusterrole.rbac.authorization.k8s.io "system:csi-external-provisioner" not found] 
  undefined
E  Unknown user "system:serviceaccount:default:csi-controller-sa"
 
  undefined

But kubectl shows the clusterrolebinding exists

$ kubectl describe clusterrolebindings csi-controller-attacher-binding
Name:         csi-controller-attacher-binding
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"name":"csi-controller-attacher-binding","namespa...
Role:
  Kind:  ClusterRole
  Name:  system:csi-external-attacher
Subjects:
  Kind            Name               Namespace
  ----            ----               ---------
  ServiceAccount  csi-controller-sa  default

$ kubectl describe clusterrolebindings csi-controller-provisioner-binding
Name:         csi-controller-provisioner-binding
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"name":"csi-controller-provisioner-binding","name...
Role:
  Kind:  ClusterRole
  Name:  system:csi-external-provisioner
Subjects:
  Kind            Name               Namespace
  ----            ----               ---------
  ServiceAccount  csi-controller-sa  default

[davidz627]

I have been unable to reproduce this issue. What version of GKE are you using?

I find it very strange that the cluster is showing the objects exist but the errors seem to be saying that they don't. And at this point everything is just done in the default namespace so I dont see that being a problem right now.

I'm going to push an update to solve: #66. And once that is in the driver should be plug and play with GKE

[saad-ali]

Version:

$ kubectl version
Client Version: version.Info{Major:"", Minor:"", GitVersion:"v0.0.0-master+$Format:%h$", GitCommit:"$Format:%H$", GitTreeState:"not a git tree", BuildDate:"1970-01-01T00:00:00Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.7-gke.3", GitCommit:"9b5b719c5f295c99de68ffb5b63101b0e0175376", GitTreeState:"clean", BuildDate:"2018-05-31T18:32:23Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

Verified that #66 is resolved with the latest commit. But I'm still running in to this, and Leonid is running in to other issues.

[msau42]

Your 1.9.7 cluster is an alpha cluster? David and I are have both tested against GKE 1.10.5

[davidz627]

The default external component roles were not introduced until 1.11, they were also cherry-picked to 1.10.5 . No 1.9 version has these cluster roles, you will have to create them yourself if using version <1.10.4

[davidz627]

I think running this on your cluster will come up empty: kubectl get clusterroles | grep csi

Weird that clusterrolebinding doesn't do any validation to make sure the cluster roles actually exist though

[davidz627]

Closing this issue as we have no repro steps, feel free to reopen if you are still seeing this issue.