Skip to content

Commit 5146607

Browse files
authored
Merge pull request #3 from bswartz/respin-release-tools
Respin release tools
2 parents 17207d4 + 29e7d31 commit 5146607

File tree

10 files changed

+120
-66
lines changed

10 files changed

+120
-66
lines changed

.cloudbuild.sh

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
release-tools/cloudbuild.sh

cloudbuild.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
release-tools/cloudbuild.yaml

release-tools/README.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,11 @@ The expected repository layout is:
2121
Dockerfile in the root when only building a single command
2222
- `Makefile` - includes `release-tools/build.make` and sets
2323
configuration variables
24-
- `.travis.yml` - a symlink to `release-tools/.travis.yml`
24+
- `.prow.sh` script which imports `release-tools/prow.sh`
25+
and may contain further customization
26+
- `.cloudbuild.sh` and `cloudbuild.yaml` as symlinks to
27+
the corresponding files in `release-tools` or (if necessary)
28+
as custom files
2529

2630
To create a release, tag a certain revision with a name that
2731
starts with `v`, for example `v1.0.0`, then `make push`

release-tools/SIDECAR_RELEASE_PROCESS.md

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,3 +104,47 @@ naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.
104104
CSI hostpath driver with the new sidecars in the [CSI repo](https://github.com/kubernetes-csi/csi-driver-host-path/tree/master/deploy)
105105
and [k/k
106106
in-tree](https://github.com/kubernetes/kubernetes/tree/master/test/e2e/testing-manifests/storage-csi/hostpath/hostpath)
107+
108+
## Adding support for a new Kubernetes release
109+
110+
1. Add the new release to `k8s_versions` in
111+
https://github.com/kubernetes/test-infra/blob/090dec5dd535d5f61b7ba52e671a810f5fc13dfd/config/jobs/kubernetes-csi/gen-jobs.sh#L25
112+
to enable generating a job for it. Set `experimental_k8s_version`
113+
in
114+
https://github.com/kubernetes/test-infra/blob/090dec5dd535d5f61b7ba52e671a810f5fc13dfd/config/jobs/kubernetes-csi/gen-jobs.sh#L40
115+
to ensure that the new jobs aren't run for PRs unless explicitly
116+
requested. Generate and submit the new jobs.
117+
1. Create a test PR to try out the new job in some repo with `/test
118+
pull-kubernetes-csi-<repo>-<x.y>-on-kubernetes-<x.y>` where x.y
119+
matches the Kubernetes release. Alternatively, run .prow.sh in that
120+
repo locally with `CSI_PROW_KUBERNETES_VERSION=x.y.z`.
121+
1. Optional: update to a [new
122+
release](https://github.com/kubernetes-sigs/kind/tags) of kind with
123+
pre-built images for the new Kubernetes release. This is optional
124+
if the current version of kind is able to build images for the new
125+
Kubernetes release. However, jobs require less resources when they
126+
don't need to build those images from the Kubernetes source code.
127+
This change needs to be tried out in a PR against a component
128+
first, then get submitted against csi-release-tools.
129+
1. Optional: propagate the updated csi-release-tools to all components
130+
with the script from
131+
https://github.com/kubernetes-csi/csi-release-tools/issues/7#issuecomment-707025402
132+
1. Once it is likely to work in all components, unset
133+
`experimental_k8s_version` and submit the updated jobs.
134+
1. Once all sidecars for the new Kubernetes release are released,
135+
either bump the version number of the images in the existing
136+
[csi-driver-host-path
137+
deployments](https://github.com/kubernetes-csi/csi-driver-host-path/tree/master/deploy)
138+
and/or create a new deployment, depending on what Kubernetes
139+
release an updated sidecar is compatible with. If no new deployment
140+
is needed, then add a symlink to document that there intentionally
141+
isn't a separate deployment. This symlink is not needed for Prow
142+
testing because that will use "kubernetes-latest" as fallback.
143+
Update that link when creating a new deployment.
144+
1. Create a new csi-driver-host-path release.
145+
1. Bump `CSI_PROW_DRIVER_VERSION` in prow.sh to that new release and
146+
(eventually) roll that change out to all repos by updating
147+
`release-tools` in them. This is used when testing manually. The
148+
Prow jobs override that value, so also update
149+
`hostpath_driver_version` in
150+
https://github.com/kubernetes/test-infra/blob/91b04e6af3a40a9bcff25aa030850a4721e2dd2b/config/jobs/kubernetes-csi/gen-jobs.sh#L46-L47

release-tools/build.make

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,7 @@ $(CMDS:%=push-multiarch-%): push-multiarch-%: check-pull-base-ref build-%
149149
--platform=$$os/$$arch \
150150
--file $$(eval echo \$${dockerfile_$$os}) \
151151
--build-arg binary=./bin/$*$$suffix \
152+
--build-arg ARCH=$$arch \
152153
--label revision=$(REV) \
153154
.; \
154155
done; \

release-tools/cloudbuild.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ options:
2525
substitution_option: ALLOW_LOOSE
2626
steps:
2727
# The image must contain bash and curl. Ideally it should also contain
28-
# the desired version of Go (currently defined in release-tools/travis.yml),
28+
# the desired version of Go (currently defined in release-tools/prow.sh),
2929
# but that just speeds up the build and is not required.
3030
- name: 'gcr.io/k8s-testimages/gcb-docker-gcloud:v20200421-a2bf5f8'
3131
entrypoint: ./.cloudbuild.sh

release-tools/go-get-kubernetes.sh

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,12 @@ mods=$( (set -x; curl --silent --show-error --fail "https://raw.githubuserconten
5555
sed -n 's|.*k8s.io/\(.*\) => ./staging/src/k8s.io/.*|k8s.io/\1|p'
5656
) || die "failed to determine Kubernetes staging modules"
5757
for mod in $mods; do
58+
if ! (env GO111MODULE=on go mod graph) | grep "$mod@" > /dev/null; then
59+
echo "Kubernetes module $mod is not used, skipping"
60+
# Remove the module from go.mod "replace" that was added by an older version of this script.
61+
(set -x; env GO111MODULE=on go mod edit "-dropreplace=$mod") || die "'go mod edit' failed"
62+
continue
63+
fi
5864
# The presence of a potentially incomplete go.mod file affects this command,
5965
# so move elsewhere.
6066
modinfo=$(set -x; cd /; env GO111MODULE=on go mod download -json "$mod@kubernetes-${k8s}") ||

release-tools/prow.sh

Lines changed: 58 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -85,37 +85,12 @@ configvar CSI_PROW_BUILD_PLATFORMS "linux amd64; windows amd64 .exe; linux ppc64
8585
# which is disabled with GOFLAGS=-mod=vendor).
8686
configvar GOFLAGS_VENDOR "$( [ -d vendor ] && echo '-mod=vendor' )" "Go flags for using the vendor directory"
8787

88-
# Go versions can be specified separately for different tasks
89-
# If the pre-installed Go is missing or a different
90-
# version, the required version here will get installed
91-
# from https://golang.org/dl/.
92-
go_from_travis_yml () {
93-
grep "^ *- go:" "${RELEASE_TOOLS_ROOT}/travis.yml" | sed -e 's/.*go: *//'
94-
}
95-
configvar CSI_PROW_GO_VERSION_BUILD "$(go_from_travis_yml)" "Go version for building the component" # depends on component's source code
88+
configvar CSI_PROW_GO_VERSION_BUILD "1.16" "Go version for building the component" # depends on component's source code
9689
configvar CSI_PROW_GO_VERSION_E2E "" "override Go version for building the Kubernetes E2E test suite" # normally doesn't need to be set, see install_e2e
9790
configvar CSI_PROW_GO_VERSION_SANITY "${CSI_PROW_GO_VERSION_BUILD}" "Go version for building the csi-sanity test suite" # depends on CSI_PROW_SANITY settings below
9891
configvar CSI_PROW_GO_VERSION_KIND "${CSI_PROW_GO_VERSION_BUILD}" "Go version for building 'kind'" # depends on CSI_PROW_KIND_VERSION below
9992
configvar CSI_PROW_GO_VERSION_GINKGO "${CSI_PROW_GO_VERSION_BUILD}" "Go version for building ginkgo" # depends on CSI_PROW_GINKGO_VERSION below
10093

101-
# kind version to use. If the pre-installed version is different,
102-
# the desired version is downloaded from https://github.com/kubernetes-sigs/kind/releases
103-
# (if available), otherwise it is built from source.
104-
configvar CSI_PROW_KIND_VERSION "v0.9.0" "kind"
105-
106-
# kind images to use. Must match the kind version.
107-
# The release notes of each kind release list the supported images.
108-
configvar CSI_PROW_KIND_IMAGES "kindest/node:v1.19.1@sha256:98cf5288864662e37115e362b23e4369c8c4a408f99cbc06e58ac30ddc721600
109-
kindest/node:v1.18.8@sha256:f4bcc97a0ad6e7abaf3f643d890add7efe6ee4ab90baeb374b4f41a4c95567eb
110-
kindest/node:v1.17.11@sha256:5240a7a2c34bf241afb54ac05669f8a46661912eab05705d660971eeb12f6555
111-
kindest/node:v1.16.15@sha256:a89c771f7de234e6547d43695c7ab047809ffc71a0c3b65aa54eda051c45ed20
112-
kindest/node:v1.15.12@sha256:d9b939055c1e852fe3d86955ee24976cab46cba518abcb8b13ba70917e6547a6
113-
kindest/node:v1.14.10@sha256:ce4355398a704fca68006f8a29f37aafb49f8fc2f64ede3ccd0d9198da910146
114-
kindest/node:v1.13.12@sha256:1c1a48c2bfcbae4d5f4fa4310b5ed10756facad0b7a2ca93c7a4b5bae5db29f5" "kind images"
115-
116-
# Use kind node-image --type=bazel by default, but allow to disable that.
117-
configvar CSI_PROW_USE_BAZEL true "use Bazel during 'kind node-image' invocation"
118-
11994
# ginkgo test runner version to use. If the pre-installed version is
12095
# different, the desired version is built from source.
12196
configvar CSI_PROW_GINKGO_VERSION v1.7.0 "Ginkgo"
@@ -152,10 +127,37 @@ configvar CSI_PROW_KUBERNETES_VERSION 1.17.0 "Kubernetes"
152127
# when a Prow job just defines the Kubernetes version.
153128
csi_prow_kubernetes_version_suffix="$(echo "${CSI_PROW_KUBERNETES_VERSION}" | tr . _ | tr '[:lower:]' '[:upper:]' | sed -e 's/^RELEASE-//' -e 's/\([0-9]*\)_\([0-9]*\).*/\1_\2/')"
154129

155-
# Work directory. It has to allow running executables, therefore /tmp
156-
# is avoided. Cleaning up after the script is intentionally left to
157-
# the caller.
158-
configvar CSI_PROW_WORK "$(mkdir -p "$GOPATH/pkg" && mktemp -d "$GOPATH/pkg/csiprow.XXXXXXXXXX")" "work directory"
130+
# Only the latest KinD is (eventually) guaranteed to work with the
131+
# latest Kubernetes. For example, KinD 0.10.0 failed with Kubernetes
132+
# 1.21.0-beta1. Therefore the default version of KinD is "main"
133+
# for that, otherwise the latest stable release for which we then
134+
# list the officially supported images below.
135+
kind_version_default () {
136+
case "${CSI_PROW_KUBERNETES_VERSION}" in
137+
latest|master)
138+
echo main;;
139+
*)
140+
echo v0.10.0;;
141+
esac
142+
}
143+
144+
# kind version to use. If the pre-installed version is different,
145+
# the desired version is downloaded from https://github.com/kubernetes-sigs/kind/releases
146+
# (if available), otherwise it is built from source.
147+
configvar CSI_PROW_KIND_VERSION "$(kind_version_default)" "kind"
148+
149+
# kind images to use. Must match the kind version.
150+
# The release notes of each kind release list the supported images.
151+
configvar CSI_PROW_KIND_IMAGES "kindest/node:v1.20.2@sha256:8f7ea6e7642c0da54f04a7ee10431549c0257315b3a634f6ef2fecaaedb19bab
152+
kindest/node:v1.19.7@sha256:a70639454e97a4b733f9d9b67e12c01f6b0297449d5b9cbbef87473458e26dca
153+
kindest/node:v1.18.15@sha256:5c1b980c4d0e0e8e7eb9f36f7df525d079a96169c8a8f20d8bd108c0d0889cc4
154+
kindest/node:v1.17.17@sha256:7b6369d27eee99c7a85c48ffd60e11412dc3f373658bc59b7f4d530b7056823e
155+
kindest/node:v1.16.15@sha256:c10a63a5bda231c0a379bf91aebf8ad3c79146daca59db816fb963f731852a99
156+
kindest/node:v1.15.12@sha256:67181f94f0b3072fb56509107b380e38c55e23bf60e6f052fbd8052d26052fb5
157+
kindest/node:v1.14.10@sha256:3fbed72bcac108055e46e7b4091eb6858ad628ec51bf693c21f5ec34578f6180" "kind images"
158+
159+
# Use kind node-image --type=bazel by default, but allow to disable that.
160+
configvar CSI_PROW_USE_BAZEL true "use Bazel during 'kind node-image' invocation"
159161

160162
# By default, this script tests sidecars with the CSI hostpath driver,
161163
# using the install_csi_driver function. That function depends on
@@ -354,10 +356,23 @@ configvar CSI_SNAPSHOTTER_VERSION "$(default_csi_snapshotter_version)" "external
354356
# to all the K8s versions we test against
355357
configvar CSI_PROW_E2E_SKIP 'Disruptive|different\s+node' "tests that need to be skipped"
356358

357-
# This is the directory for additional result files. Usually set by Prow, but
358-
# if not (for example, when invoking manually) it defaults to the work directory.
359-
configvar ARTIFACTS "${CSI_PROW_WORK}/artifacts" "artifacts"
360-
mkdir -p "${ARTIFACTS}"
359+
# This creates directories that are required for testing.
360+
ensure_paths () {
361+
# Work directory. It has to allow running executables, therefore /tmp
362+
# is avoided. Cleaning up after the script is intentionally left to
363+
# the caller.
364+
configvar CSI_PROW_WORK "$(mkdir -p "$GOPATH/pkg" && mktemp -d "$GOPATH/pkg/csiprow.XXXXXXXXXX")" "work directory"
365+
366+
# This is the directory for additional result files. Usually set by Prow, but
367+
# if not (for example, when invoking manually) it defaults to the work directory.
368+
configvar ARTIFACTS "${CSI_PROW_WORK}/artifacts" "artifacts"
369+
mkdir -p "${ARTIFACTS}"
370+
371+
# For additional tools.
372+
CSI_PROW_BIN="${CSI_PROW_WORK}/bin"
373+
mkdir -p "${CSI_PROW_BIN}"
374+
PATH="${CSI_PROW_BIN}:$PATH"
375+
}
361376

362377
run () {
363378
echo "$(date) $(go version | sed -e 's/.*version \(go[^ ]*\).*/\1/') $(if [ "$(pwd)" != "${REPO_DIR}" ]; then pwd; fi)\$" "$@" >&2
@@ -377,11 +392,6 @@ die () {
377392
exit 1
378393
}
379394

380-
# For additional tools.
381-
CSI_PROW_BIN="${CSI_PROW_WORK}/bin"
382-
mkdir -p "${CSI_PROW_BIN}"
383-
PATH="${CSI_PROW_BIN}:$PATH"
384-
385395
# Ensure that PATH has the desired version of the Go tools, then run command given as argument.
386396
# Empty parameter uses the already installed Go. In Prow, that version is kept up-to-date by
387397
# bumping the container image regularly.
@@ -410,7 +420,7 @@ install_kind () {
410420
chmod u+x "${CSI_PROW_WORK}/bin/kind"
411421
else
412422
git_checkout https://github.com/kubernetes-sigs/kind "${GOPATH}/src/sigs.k8s.io/kind" "${CSI_PROW_KIND_VERSION}" --depth=1 &&
413-
(cd "${GOPATH}/src/sigs.k8s.io/kind" && make install INSTALL_DIR="${CSI_PROW_WORK}/bin")
423+
(cd "${GOPATH}/src/sigs.k8s.io/kind" && run_with_go "$CSI_PROW_GO_VERSION_KIND" make install INSTALL_DIR="${CSI_PROW_WORK}/bin")
414424
fi
415425
}
416426

@@ -576,7 +586,8 @@ start_cluster () {
576586

577587
go_version="$(go_version_for_kubernetes "${CSI_PROW_WORK}/src/kubernetes" "$version")" || die "cannot proceed without knowing Go version for Kubernetes"
578588
# Changing into the Kubernetes source code directory is a workaround for https://github.com/kubernetes-sigs/kind/issues/1910
579-
(cd "${CSI_PROW_WORK}/src/kubernetes" && run_with_go "$go_version" kind build node-image --image csiprow/node:latest --type="$type" --kube-root "${CSI_PROW_WORK}/src/kubernetes") || die "'kind build node-image' failed"
589+
# shellcheck disable=SC2046
590+
(cd "${CSI_PROW_WORK}/src/kubernetes" && run_with_go "$go_version" kind build node-image --image csiprow/node:latest $(if [ "$CSI_PROW_KIND_VERSION" != "main" ]; then echo --type="$type"; fi) --kube-root "${CSI_PROW_WORK}/src/kubernetes") || die "'kind build node-image' failed"
580591
csi_prow_kind_have_kubernetes=true
581592
fi
582593
image="csiprow/node:latest"
@@ -1090,6 +1101,9 @@ main () {
10901101
local images ret
10911102
ret=0
10921103
1104+
# Set up work directory.
1105+
ensure_paths
1106+
10931107
images=
10941108
if ${CSI_PROW_BUILD_JOB}; then
10951109
# A successful build is required for testing.
@@ -1251,6 +1265,9 @@ gcr_cloud_build () {
12511265
# Required for "docker buildx build --push".
12521266
gcloud auth configure-docker
12531267
1268+
# Might not be needed here, but call it just in case.
1269+
ensure_paths
1270+
12541271
if find . -name Dockerfile | grep -v ^./vendor | xargs --no-run-if-empty cat | grep -q ^RUN; then
12551272
# Needed for "RUN" steps on non-linux/amd64 platforms.
12561273
# See https://github.com/multiarch/qemu-user-static#getting-started

release-tools/travis.yml

Lines changed: 0 additions & 21 deletions
This file was deleted.

release-tools/verify-go-version.sh

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,9 @@ die () {
2929
version=$("$GO" version) || die "determining version of $GO failed"
3030
# shellcheck disable=SC2001
3131
majorminor=$(echo "$version" | sed -e 's/.*go\([0-9]*\)\.\([0-9]*\).*/\1.\2/')
32-
# shellcheck disable=SC2001
33-
expected=$(grep "^ *- go:" "release-tools/travis.yml" | sed -e 's/.*go: *\([0-9]*\)\.\([0-9]*\).*/\1.\2/')
32+
# SC1091: Not following: release-tools/prow.sh was not specified as input (see shellcheck -x).
33+
# shellcheck disable=SC1091
34+
expected=$(. release-tools/prow.sh >/dev/null && echo "$CSI_PROW_GO_VERSION_BUILD")
3435

3536
if [ "$majorminor" != "$expected" ]; then
3637
cat >&2 <<EOF

0 commit comments

Comments
 (0)