Integration test failure due to missing layer

[pwschuurman]

Example failing run:

https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/1212/pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration/1656011148585078784/build-log.txt

Example passing run:

https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/1212/pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration/1651003375631732736/build-log.txt

Data for posterity: https://gist.github.com/pwschuurman/99776f47c24ec99e1889310dbdfede05

Error from failing run:

I0509 19:24:16.208] #1 [internal] load build definition from Dockerfile
I0509 19:24:17.063] #1 transferring dockerfile:
I0509 19:24:17.064] #1 transferring dockerfile: 765B done
I0509 19:24:17.065] #1 DONE 0.1s
I0509 19:24:17.065] 
I0509 19:24:17.066] #2 [internal] load .dockerignore
I0509 19:24:17.066] #2 transferring context: 2B done
I0509 19:24:17.066] #2 DONE 0.1s
I0509 19:24:17.066] 
I0509 19:24:17.066] #3 [internal] load metadata for registry.k8s.io/build-image/go-runner:v2.3.1-go1.20.4-bullseye.0
I0509 19:24:17.066] #3 DONE 0.1s
I0509 19:24:17.066] 
I0509 19:24:17.066] #4 [1/2] FROM registry.k8s.io/build-image/go-runner:v2.3.1-go1.20.4-bullseye.0@sha256:c3b50b00882c6115bf43c2ab2300bd1ae433a13a8eed8bea5ceafddb204d332d
I0509 19:24:17.066] #4 resolve registry.k8s.io/build-image/go-runner:v2.3.1-go1.20.4-bullseye.0@sha256:c3b50b00882c6115bf43c2ab2300bd1ae433a13a8eed8bea5ceafddb204d332d
I0509 19:24:17.067] #4 resolve registry.k8s.io/build-image/go-runner:v2.3.1-go1.20.4-bullseye.0@sha256:c3b50b00882c6115bf43c2ab2300bd1ae433a13a8eed8bea5ceafddb204d332d 0.2s done
I0509 19:24:17.067] #4 sha256:82294f290df24d1323e010e58141b74d36dbcc0b507df2b3d3a7213c91b18738 723.70kB / 723.70kB 0.1s done
I0509 19:24:17.067] #4 DONE 1.9s
I0509 19:24:17.068] 
I0509 19:24:17.070] #5 [internal] load build context
I0509 19:24:17.070] #5 ...
I0509 19:24:17.071] 
I0509 19:24:17.071] #4 [1/2] FROM registry.k8s.io/build-image/go-runner:v2.3.1-go1.20.4-bullseye.0@sha256:c3b50b00882c6115bf43c2ab2300bd1ae433a13a8eed8bea5ceafddb204d332d
I0509 19:24:17.072] #4 extracting sha256:82294f290df24d1323e010e58141b74d36dbcc0b507df2b3d3a7213c91b18738 0.3s done
I0509 19:24:17.072] #4 DONE 2.2s
I0509 19:24:17.072] 
I0509 19:24:17.072] #5 [internal] load build context
I0509 19:24:17.072] #5 transferring context: 49.32MB 4.7s done
I0509 19:24:17.072] #5 DONE 4.9s
I0509 19:24:17.072] 
I0509 19:24:17.072] #6 [2/2] COPY  /bin/
I0509 19:24:17.072] #6 DONE 5.1s
I0509 19:24:17.073] 
I0509 19:24:17.073] #7 exporting to oci image format
I0509 19:24:17.073] #7 exporting layers
I0509 19:24:17.073] #7 exporting layers 5.9s done
I0509 19:24:17.073] #7 exporting manifest sha256:61a0b5cb1499af9ccc5d63e984029e4862e1b54c49eb46d6042733bc8d087fbb 0.0s done
I0509 19:24:17.073] #7 exporting config sha256:30227d1f47eca6754961b661497e7550baf29b42f2701190c39154ecdb450227 0.0s done
I0509 19:24:17.073] #7 sending tarball
I0509 19:24:17.073] #7 sending tarball 1.4s done
I0509 19:24:17.073] #7 ERROR: failed to get reader: content digest sha256:b4ca4c215f483111b64ec6919f1659ff475d7080a649d6acd78a6ade562a4a63: not found

This may be due to https://github.com/moby/buildkit/releases, which is fixed in 0.11.3 version of buildkit. We can try to update the buildkit version, to pick up this change, and see if this reproduces. From what I can tell it's a transient error.

[pwschuurman]

Looks like this is failing on 0.10 and 0.9 branches as well

for branch in 10 9; do ~/go/src/k8s.io/kubernetes/hack/cherry_pick_pull.sh upstream/release-1.${branch} 1218; done

• release-1.9: Automated cherry pick of #1218: Update buildkit version from 0.10.6 -> 0.11.3 #1226
• release-1.10: Automated cherry pick of #1218: Update buildkit version from 0.10.6 -> 0.11.3 #1225