Failed to set up metadata service: failed to get machine-type: metadata: GCE metadata "instance/machine-type" not defined

[philipstromback]

Hi,
I have a problem with installing the driver any version above v0.5.0 because it have problem to get the instance-type from the nodes. I'm running in a region cluster with a n1-standard-2 node-pool.

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.7-gke.10", GitCommit:"8cea5f8ae165065f0d35e5de5dfa2f73617f02d1", GitTreeState:"clean", BuildDate:"2019-10-05T00:08:10Z", GoVersion:"go1.12.9b4", Compiler:"gc", Platform:"linux/amd64"}

Releases:
0.5.1
0.5.2
0.6.0

Errors:

kubectl logs -n gce-pd-csi-driver csi-gce-pd-controller-0 gce-pd-driver
I1025 07:26:33.786811       1 main.go:60] Driver vendor version v0.6.0-gke.0
I1025 07:26:33.786885       1 gce.go:81] Using GCE provider config <nil>
I1025 07:26:33.787420       1 gce.go:133] GOOGLE_APPLICATION_CREDENTIALS env var set /etc/cloud-sa/cloud-sa.json
I1025 07:26:33.787594       1 gce.go:137] Using DefaultTokenSource &oauth2.reuseTokenSource{new:jwt.jwtSource{ctx:(*context.emptyCtx)(0xc000080010), conf:(*jwt.Config)(0xc000260280)}, mu:sync.Mutex{state:0, sema:0x0}, t:(*oauth2.Token)(nil)}
I1025 07:26:34.014961       1 gce.go:222] Using GCP project ID from the Metadata server: "test123"
I1025 07:26:34.015174       1 mount_linux.go:160] Detected OS without systemd
F1025 07:26:34.016646       1 main.go:75] Failed to set up metadata service: failed to get machine-type: metadata: GCE metadata "instance/machine-type" not defined

Any idea what's wrong and how to solved it?

Regards
Philip

[philipstromback]

After more investigation it seems to have to do that I use workload identity in the cluster which seems not to pass the instance-type to the pods.

[msau42]

/kind bug
/assign @davidz627

[davidz627]

Hi @philipstromback, thanks for bringing this up and adding context with regards to workload identity. We are investigating now

[hilliao]

I have workload identity enabled on a new cluster created only for testing the driver. I have a similar error message with difference of GOOGLE_APPLICATION_CREDENTIALS env var not set. I tested a virtually identical cluster with workload identity disabled without such problem.

hil@xeon:~$ kubectl logs  csi-gce-pd-node-ztq77 gce-pd-driver -n gce-pd-csi-driver
I1108 19:58:40.792495       1 main.go:60] Driver vendor version v0.6.0-gke.0
I1108 19:58:40.792562       1 gce.go:81] Using GCE provider config <nil>
W1108 19:58:40.797720       1 gce.go:135] GOOGLE_APPLICATION_CREDENTIALS env var not set
I1108 19:58:40.801630       1 gce.go:137] Using DefaultTokenSource &oauth2.reuseTokenSource{new:google.computeSource{account:"", scopes:[]string{"https://www.googleapis.com/auth/cloud-platform", "https://www.googleapis.com/auth/compute"}}, mu:sync.Mutex{state:0, sema:0x0}, t:(*oauth2.Token)(nil)}
I1108 19:58:40.859967       1 gce.go:222] Using GCP project ID from the Metadata server: "tcsplayground"
I1108 19:58:40.860317       1 mount_linux.go:160] Detected OS without systemd
F1108 19:58:40.862059       1 main.go:75] Failed to set up metadata service: failed to get machine-type: metadata: GCE metadata "instance/machine-type" not defined

[davidz627]

Hi All,

To workaround this issue please run both the controller and node deployments with hostNetwork:true
These YAMLs can be found here: https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/tree/master/deploy/kubernetes/base

We are working on a longer term fix to lift the restriction of using hostNetwork:true when the driver is used in a cluster with Workload Identity.

Thank you for bringing up this issue. Leaving it open for others to see the workaround as well as to track progress of longer term fix.

[hilliao]

@davidz627 , it's not obvious how users can read driver-install and identify which .yaml file to put hostNetwork:true. Can you give more specific instructions on which yaml files to modify and which line to put hostNetwork:true before running through driver-install?

[davidz627]

@hilliao we will actually just update the deployments to work and make some patch releases. Please check out the linked PR's if you want to use it earlier

[hilliao]

Does it mean devops can just follow driver-install with workload identity enabled on the cluster without encounter the error in the issue title?