From 0535c0d4f101a4c147a710c5fff74e81368dc7e3 Mon Sep 17 00:00:00 2001 From: Aaron Crickenberger Date: Wed, 16 Jan 2019 10:47:35 -0800 Subject: [PATCH 001/107] Initial commit from kubernetes-template-project --- CONTRIBUTING.md | 31 +++++++ LICENSE | 201 +++++++++++++++++++++++++++++++++++++++++++++ OWNERS | 6 ++ OWNERS_ALIASES | 20 +++++ README.md | 29 +++++++ RELEASE.md | 9 ++ SECURITY_CONTACTS | 14 ++++ code-of-conduct.md | 3 + 8 files changed, 313 insertions(+) create mode 100644 CONTRIBUTING.md create mode 100644 LICENSE create mode 100644 OWNERS create mode 100644 OWNERS_ALIASES create mode 100644 README.md create mode 100644 RELEASE.md create mode 100644 SECURITY_CONTACTS create mode 100644 code-of-conduct.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..de47115 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,31 @@ +# Contributing Guidelines + +Welcome to Kubernetes. We are excited about the prospect of you joining our [community](https://github.com/kubernetes/community)! The Kubernetes community abides by the CNCF [code of conduct](code-of-conduct.md). Here is an excerpt: + +_As contributors and maintainers of this project, and in the interest of fostering an open and welcoming community, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities._ + +## Getting Started + +We have full documentation on how to get started contributing here: + + + +- [Contributor License Agreement](https://git.k8s.io/community/CLA.md) Kubernetes projects require that you sign a Contributor License Agreement (CLA) before we can accept your pull requests +- [Kubernetes Contributor Guide](http://git.k8s.io/community/contributors/guide) - Main contributor documentation, or you can just jump directly to the [contributing section](http://git.k8s.io/community/contributors/guide#contributing) +- [Contributor Cheat Sheet](https://git.k8s.io/community/contributors/guide/contributor-cheatsheet.md) - Common resources for existing developers + +## Mentorship + +- [Mentoring Initiatives](https://git.k8s.io/community/mentoring) - We have a diverse set of mentorship programs available that are always looking for volunteers! + + diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..8dada3e --- /dev/null +++ b/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright {yyyy} {name of copyright owner} + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/OWNERS b/OWNERS new file mode 100644 index 0000000..e4c79b9 --- /dev/null +++ b/OWNERS @@ -0,0 +1,6 @@ +# See the OWNERS docs: https://git.k8s.io/community/contributors/guide/owners.md + +approvers: + # TODO: in your repo created from this template, you should replace the + # steering-committee with a list of project owners, see the doc linked above. + - steering-committee diff --git a/OWNERS_ALIASES b/OWNERS_ALIASES new file mode 100644 index 0000000..8bcd3a6 --- /dev/null +++ b/OWNERS_ALIASES @@ -0,0 +1,20 @@ +# See the OWNERS_ALIASES docs: https://git.k8s.io/community/contributors/guide/owners.md#owners_aliases + +aliases: + # TODO: remove this alias, it will go stale in your repo, and in your repo + # you should have your own set of approvers (see OWNERS) + # in the original template repo, we must maintain this list to approve changes + # to the template itself + steering-committee: + - bgrant0607 + - brendandburns + - derekwaynecarr + - dims + - jbeda + - michelleN + - philips + - pwittrock + - sarahnovotny + - smarterclayton + - spiffxp + - timothysc diff --git a/README.md b/README.md new file mode 100644 index 0000000..9431b7b --- /dev/null +++ b/README.md @@ -0,0 +1,29 @@ +# Kubernetes Template Project + +The Kubernetes Template Project is a template for starting new projects in the GitHub organizations owned by Kubernetes. All Kubernetes projects, at minimum, must have the following files: + +- a `README.md` outlining the project goals, sponsoring sig, and community contact information +- an `OWNERS` with the project leads listed as approvers ([docs on `OWNERS` files][owners]) +- a `CONTRIBUTING.md` outlining how to contribute to the project +- an unmodified copy of `code-of-conduct.md` from this repo, which outlines community behavior and the consequences of breaking the code +- a `LICENSE` which must be Apache 2.0 for code projects, or [Creative Commons 4.0] for documentation repositories, without any custom content +- a `SECURITY_CONTACTS` with the contact points for the Product Security Team + to reach out to for triaging and handling of incoming issues. They must agree to abide by the + [Embargo Policy](https://github.com/kubernetes/sig-release/blob/master/security-release-process-documentation/security-release-process.md#embargo-policy) + and will be removed and replaced if they violate that agreement. + +## Community, discussion, contribution, and support + +Learn how to engage with the Kubernetes community on the [community page](http://kubernetes.io/community/). + +You can reach the maintainers of this project at: + +- [Slack](http://slack.k8s.io/) +- [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-dev) + +### Code of conduct + +Participation in the Kubernetes community is governed by the [Kubernetes Code of Conduct](code-of-conduct.md). + +[owners]: https://git.k8s.io/community/contributors/guide/owners.md +[Creative Commons 4.0]: https://git.k8s.io/website/LICENSE diff --git a/RELEASE.md b/RELEASE.md new file mode 100644 index 0000000..7274b34 --- /dev/null +++ b/RELEASE.md @@ -0,0 +1,9 @@ +# Release Process + +The Kubernetes Template Project is released on an as-needed basis. The process is as follows: + +1. An issue is proposing a new release with a changelog since the last release +1. All [OWNERS](OWNERS) must LGTM this release +1. An OWNER runs `git tag -s $VERSION` and inserts the changelog and pushes the tag with `git push $VERSION` +1. The release issue is closed +1. An announcement email is sent to `kubernetes-dev@googlegroups.com` with the subject `[ANNOUNCE] kubernetes-template-project $VERSION is released` diff --git a/SECURITY_CONTACTS b/SECURITY_CONTACTS new file mode 100644 index 0000000..3968917 --- /dev/null +++ b/SECURITY_CONTACTS @@ -0,0 +1,14 @@ +# Defined below are the security contacts for this repo. +# +# They are the contact point for the Product Security Team to reach out +# to for triaging and handling of incoming issues. +# +# The below names agree to abide by the +# [Embargo Policy](https://github.com/kubernetes/sig-release/blob/master/security-release-process-documentation/security-release-process.md#embargo-policy) +# and will be removed and replaced if they violate that agreement. +# +# DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE +# INSTRUCTIONS AT https://kubernetes.io/security/ + +bob +alice diff --git a/code-of-conduct.md b/code-of-conduct.md new file mode 100644 index 0000000..0d15c00 --- /dev/null +++ b/code-of-conduct.md @@ -0,0 +1,3 @@ +# Kubernetes Community Code of Conduct + +Please refer to our [Kubernetes Community Code of Conduct](https://git.k8s.io/community/code-of-conduct.md) From b41cac350d9dd1b9c478b16c398745aeb95390a6 Mon Sep 17 00:00:00 2001 From: Aaron Crickenberger Date: Wed, 16 Jan 2019 10:51:52 -0800 Subject: [PATCH 002/107] Add files from github.com/pohly/csi-build-rules --- README.md | 79 ++-- build.make | 1211 ++++++++++++++++++++++++++++++++++++++++++++++++++++ travis.yml | 14 + 3 files changed, 1275 insertions(+), 29 deletions(-) create mode 100644 build.make create mode 100644 travis.yml diff --git a/README.md b/README.md index 9431b7b..c67e930 100644 --- a/README.md +++ b/README.md @@ -1,29 +1,50 @@ -# Kubernetes Template Project - -The Kubernetes Template Project is a template for starting new projects in the GitHub organizations owned by Kubernetes. All Kubernetes projects, at minimum, must have the following files: - -- a `README.md` outlining the project goals, sponsoring sig, and community contact information -- an `OWNERS` with the project leads listed as approvers ([docs on `OWNERS` files][owners]) -- a `CONTRIBUTING.md` outlining how to contribute to the project -- an unmodified copy of `code-of-conduct.md` from this repo, which outlines community behavior and the consequences of breaking the code -- a `LICENSE` which must be Apache 2.0 for code projects, or [Creative Commons 4.0] for documentation repositories, without any custom content -- a `SECURITY_CONTACTS` with the contact points for the Product Security Team - to reach out to for triaging and handling of incoming issues. They must agree to abide by the - [Embargo Policy](https://github.com/kubernetes/sig-release/blob/master/security-release-process-documentation/security-release-process.md#embargo-policy) - and will be removed and replaced if they violate that agreement. - -## Community, discussion, contribution, and support - -Learn how to engage with the Kubernetes community on the [community page](http://kubernetes.io/community/). - -You can reach the maintainers of this project at: - -- [Slack](http://slack.k8s.io/) -- [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-dev) - -### Code of conduct - -Participation in the Kubernetes community is governed by the [Kubernetes Code of Conduct](code-of-conduct.md). - -[owners]: https://git.k8s.io/community/contributors/guide/owners.md -[Creative Commons 4.0]: https://git.k8s.io/website/LICENSE +# [csi-build-rules](https://github.com/kubernetes-csi/csi-build-rules) + +These build and test rules can be shared between different Go projects +without modifications. Customization for the different projects happen +in the top-level Makefile. + +The rules include support for building and pushing Docker images, with +the following features: + - one or more command and image per project + - push canary and/or tagged release images + - automatically derive the image tag(s) from repo tags + - the source code revision is stored in a "revision" image label + - never overwrites an existing release image + +Usage +----- + +The expected repository layout is: + - `cmd/*/*.go` - source code for each command + - `cmd/*/Dockerfile` - docker file for each command or + Dockerfile in the root when only building a single command + - `Makefile` - includes `build-rules/build.make` and sets + configuration variables + - `.travis.yml` - a symlink to `build-rules/.travis.yml` + +To create a release, tag a certain revision with a name that +starts with `v`, for example `v1.0.0`, then `make push` +while that commit is checked out. + +It does not matter on which branch that revision exists, i.e. it is +possible to create releases directly from master. A release branch can +still be created for maintenance releases later if needed. + +Release branches are expected to be named `release-x.y` for releases +`x.y.z`. Building from such a branch creates `x.y-canary` +images. Building from master creates the main `canary` image. + +Sharing and updating +-------------------- + +[`git subtree`](https://github.com/git/git/blob/master/contrib/subtree/git-subtree.txt) +is the recommended way of maintaining a copy of the rules inside the +`build-rules` directory of a project. This way, it is possible to make +changes also locally, test them and then push them back to the shared +repository at a later time. + +Cheat sheet: + +- `git subtree pull --prefix=build-rules https://github.com/kubernetes-csi/csi-build-rules.git master` - update local copy to latest upstream +- edit, `git commit`, `git subtree push --prefix=build-rules git@github.com:/csi-build-rules.git ` - push to a new branch before submitting a PR diff --git a/build.make b/build.make new file mode 100644 index 0000000..460ee71 --- /dev/null +++ b/build.make @@ -0,0 +1,1211 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + csi-build-rules/build.make at master · pohly/csi-build-rules · GitHub + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ Skip to content +
+ + + + + + + + + +
+ +
+ +
+ +
+ + + +
+
+
+ + + + + + + + + +
+
+ + + +

+ + /csi-build-rules + +

+ +
+ + + + +
+ +
+
+ + + + + + + + Permalink + + + + + + +
+ +
+ + +
+
+
+
+ +
+
+
+
+ +
+ + Find file + + + Copy path + +
+ +
+ + + +
+ + + c03f108 + + Jan 9, 2019 + + + +
+ +
+ + + 1 contributor + + +
+ +

Users who have contributed to this file

+
+ + + + +
+ +
+
+ + + + +
+ +
+ +
+ + +
+ Raw + Blame + History +
+ + + + +
+ +
+ 100 lines (83 sloc) + + 3.78 KB +
+
+ + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
+
.PHONY: build-% build container-% container push-% push clean test
+
# A space-separated list of all commands in the repository, must be
# set in main Makefile of a repository.
# CMDS=
+
# This is the default. It can be overridden in the main Makefile after
# including build.make.
REGISTRY_NAME=quay.io/k8scsi
+
# Revision that gets built into each binary via the main.version
# string. Uses the `git describe` output based on the most recent
# version tag with a short revision suffix or, if nothing has been
# tagged yet, just the revision.
#
# Beware that tags may also be missing in shallow clones as done by
# some CI systems (like TravisCI, which pulls only 50 commits).
REV=$(shell git describe --long --tags --match='v*' --dirty 2>/dev/null || git rev-list -n1 HEAD)
+
# A space-separated list of image tags under which the current build is to be pushed.
# Determined dynamically.
IMAGE_TAGS=
+
# A "canary" image gets built if the current commit is the head of the remote "master" branch.
# That branch does not exist when building some other branch in TravisCI.
IMAGE_TAGS+=$(shell if [ "$$(git rev-list -n1 HEAD)" = "$$(git rev-list -n1 origin/master 2>/dev/null)" ]; then echo "canary"; fi)
+
# A "X.Y.Z-canary" image gets built if the current commit is the head of a "origin/release-X.Y.Z" branch.
# The actual suffix does not matter, only the "release-" prefix is checked.
IMAGE_TAGS+=$(shell git branch -r --points-at=HEAD | grep 'origin/release-' | grep -v -e ' -> ' | sed -e 's;.*/release-\(.*\);\1-canary;')
+
# A release image "vX.Y.Z" gets built if there is a tag of that format for the current commit.
# --abbrev=0 suppresses long format, only showing the closest tag.
IMAGE_TAGS+=$(shell tagged="$$(git describe --tags --match='v*' --abbrev=0)"; if [ "$$tagged" ] && [ "$$(git rev-list -n1 HEAD)" = "$$(git rev-list -n1 $$tagged)" ]; then echo $$tagged; fi)
+
# Images are named after the command contained in them.
IMAGE_NAME=$(REGISTRY_NAME)/$*
+
ifdef V
TESTARGS = -v -args -alsologtostderr -v 5
else
TESTARGS =
endif
+
build-%:
mkdir -p bin
CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$* ./cmd/$*
+
container-%: build-%
docker build -t $*:latest -f $(shell if [ -e ./cmd/$*/Dockerfile ]; then echo ./cmd/$*/Dockerfile; else echo Dockerfile; fi) --label revision=$(REV) .
+
push-%: container-%
set -ex; \
push_image () { \
docker tag $*:latest $(IMAGE_NAME):$$tag; \
docker push $(IMAGE_NAME):$$tag; \
}; \
for tag in $(IMAGE_TAGS); do \
if echo $$tag | grep -q -e '-canary$$'; then \
: "creating or overwriting canary image"; \
push_image; \
elif docker pull $(IMAGE_NAME):$$tag 2>&1 | tee /dev/stderr | grep -q "manifest for $(IMAGE_NAME):$$tag not found"; then \
: "creating release image"; \
push_image; \
else \
: "release image $(IMAGE_NAME):$$tag already exists, skipping push"; \
fi; \
done
+
build: $(CMDS:%=build-%)
container: $(CMDS:%=container-%)
push: $(CMDS:%=push-%)
+
clean:
-rm -rf bin
+
test:
go test `go list ./... | grep -v 'vendor'` $(TESTARGS)
go vet `go list ./... | grep -v vendor`
files=$$(find . -name '*.go' | grep -v './vendor'); \
if [ $$(gofmt -d $$files | wc -l) -ne 0 ]; then \
echo "formatting errors:"; \
gofmt -d $$files; \
false; \
fi
+ + + +
+ +
+ + + +
+ + +
+ + +
 +
+ + + +
+
+
+ +
+
+ +
+ + + + + + +
+ + + You can’t perform that action at this time. +
+ + + + + + + + + +
+ + You signed in with another tab or window. Reload to refresh your session. + You signed out in another tab or window. Reload to refresh your session. +
+ + + + + + +
+ Press h to open a hovercard with more details. +
+ +
+ + + + diff --git a/travis.yml b/travis.yml new file mode 100644 index 0000000..a44b001 --- /dev/null +++ b/travis.yml @@ -0,0 +1,14 @@ +language: go +sudo: required +services: + - docker +matrix: + include: + - go: 1.11.1 +script: +- make all test +after_success: + - if [ "${TRAVIS_PULL_REQUEST}" == "false" ]; then + docker login -u "${DOCKER_USERNAME}" -p "${DOCKER_PASSWORD}" quay.io; + make push; + fi From d23a16cb744734b468ac4e226c98ca8aaaea9125 Mon Sep 17 00:00:00 2001 From: Aaron Crickenberger Date: Wed, 16 Jan 2019 11:19:16 -0800 Subject: [PATCH 003/107] Port over information from kubernetes/org --- OWNERS | 11 ++++++++--- OWNERS_ALIASES | 20 -------------------- RELEASE.md | 8 +------- SECURITY_CONTACTS | 4 ++-- 4 files changed, 11 insertions(+), 32 deletions(-) delete mode 100644 OWNERS_ALIASES diff --git a/OWNERS b/OWNERS index e4c79b9..6d2f474 100644 --- a/OWNERS +++ b/OWNERS @@ -1,6 +1,11 @@ # See the OWNERS docs: https://git.k8s.io/community/contributors/guide/owners.md approvers: - # TODO: in your repo created from this template, you should replace the - # steering-committee with a list of project owners, see the doc linked above. - - steering-committee +- saad-ali +- msau42 +- pohly + +reviewers: +- saad-ali +- msau42 +- pohly diff --git a/OWNERS_ALIASES b/OWNERS_ALIASES deleted file mode 100644 index 8bcd3a6..0000000 --- a/OWNERS_ALIASES +++ /dev/null @@ -1,20 +0,0 @@ -# See the OWNERS_ALIASES docs: https://git.k8s.io/community/contributors/guide/owners.md#owners_aliases - -aliases: - # TODO: remove this alias, it will go stale in your repo, and in your repo - # you should have your own set of approvers (see OWNERS) - # in the original template repo, we must maintain this list to approve changes - # to the template itself - steering-committee: - - bgrant0607 - - brendandburns - - derekwaynecarr - - dims - - jbeda - - michelleN - - philips - - pwittrock - - sarahnovotny - - smarterclayton - - spiffxp - - timothysc diff --git a/RELEASE.md b/RELEASE.md index 7274b34..491044d 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -1,9 +1,3 @@ # Release Process -The Kubernetes Template Project is released on an as-needed basis. The process is as follows: - -1. An issue is proposing a new release with a changelog since the last release -1. All [OWNERS](OWNERS) must LGTM this release -1. An OWNER runs `git tag -s $VERSION` and inserts the changelog and pushes the tag with `git push $VERSION` -1. The release issue is closed -1. An announcement email is sent to `kubernetes-dev@googlegroups.com` with the subject `[ANNOUNCE] kubernetes-template-project $VERSION is released` +TODO: describe the release process for this project diff --git a/SECURITY_CONTACTS b/SECURITY_CONTACTS index 3968917..2af1414 100644 --- a/SECURITY_CONTACTS +++ b/SECURITY_CONTACTS @@ -10,5 +10,5 @@ # DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE # INSTRUCTIONS AT https://kubernetes.io/security/ -bob -alice +saad-ali +msau42 From a1136470c29f970e0af87d352a1b3cb1520d09dc Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Mon, 21 Jan 2019 10:08:51 +0100 Subject: [PATCH 004/107] build.make: initial content The repo was created with an HTML version of the build.make file from https://github.com/pohly/csi-build-rules/. Here's the raw file. --- build.make | 1310 ++++------------------------------------------------ 1 file changed, 99 insertions(+), 1211 deletions(-) diff --git a/build.make b/build.make index 460ee71..4214648 100644 --- a/build.make +++ b/build.make @@ -1,1211 +1,99 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - csi-build-rules/build.make at master · pohly/csi-build-rules · GitHub - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- Skip to content -
- - - - - - - - - -
- -
- -
- -
- - - -
-
-
- - - - - - - - - -
-
- - - -

- - /csi-build-rules - -

- -
- - - - -
- -
-
- - - - - - - - Permalink - - - - - - -
- -
- - -
-
-
-
- -
-
-
-
- -
- - Find file - - - Copy path - -
- -
- - - -
- - - c03f108 - - Jan 9, 2019 - - - -
- -
- - - 1 contributor - - -
- -

Users who have contributed to this file

-
- - - - -
- -
-
- - - - -
- -
- -
- - -
- Raw - Blame - History -
- - - - -
- -
- 100 lines (83 sloc) - - 3.78 KB -
-
- - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-
.PHONY: build-% build container-% container push-% push clean test
-
# A space-separated list of all commands in the repository, must be
# set in main Makefile of a repository.
# CMDS=
-
# This is the default. It can be overridden in the main Makefile after
# including build.make.
REGISTRY_NAME=quay.io/k8scsi
-
# Revision that gets built into each binary via the main.version
# string. Uses the `git describe` output based on the most recent
# version tag with a short revision suffix or, if nothing has been
# tagged yet, just the revision.
#
# Beware that tags may also be missing in shallow clones as done by
# some CI systems (like TravisCI, which pulls only 50 commits).
REV=$(shell git describe --long --tags --match='v*' --dirty 2>/dev/null || git rev-list -n1 HEAD)
-
# A space-separated list of image tags under which the current build is to be pushed.
# Determined dynamically.
IMAGE_TAGS=
-
# A "canary" image gets built if the current commit is the head of the remote "master" branch.
# That branch does not exist when building some other branch in TravisCI.
IMAGE_TAGS+=$(shell if [ "$$(git rev-list -n1 HEAD)" = "$$(git rev-list -n1 origin/master 2>/dev/null)" ]; then echo "canary"; fi)
-
# A "X.Y.Z-canary" image gets built if the current commit is the head of a "origin/release-X.Y.Z" branch.
# The actual suffix does not matter, only the "release-" prefix is checked.
IMAGE_TAGS+=$(shell git branch -r --points-at=HEAD | grep 'origin/release-' | grep -v -e ' -> ' | sed -e 's;.*/release-\(.*\);\1-canary;')
-
# A release image "vX.Y.Z" gets built if there is a tag of that format for the current commit.
# --abbrev=0 suppresses long format, only showing the closest tag.
IMAGE_TAGS+=$(shell tagged="$$(git describe --tags --match='v*' --abbrev=0)"; if [ "$$tagged" ] && [ "$$(git rev-list -n1 HEAD)" = "$$(git rev-list -n1 $$tagged)" ]; then echo $$tagged; fi)
-
# Images are named after the command contained in them.
IMAGE_NAME=$(REGISTRY_NAME)/$*
-
ifdef V
TESTARGS = -v -args -alsologtostderr -v 5
else
TESTARGS =
endif
-
build-%:
mkdir -p bin
CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$* ./cmd/$*
-
container-%: build-%
docker build -t $*:latest -f $(shell if [ -e ./cmd/$*/Dockerfile ]; then echo ./cmd/$*/Dockerfile; else echo Dockerfile; fi) --label revision=$(REV) .
-
push-%: container-%
set -ex; \
push_image () { \
docker tag $*:latest $(IMAGE_NAME):$$tag; \
docker push $(IMAGE_NAME):$$tag; \
}; \
for tag in $(IMAGE_TAGS); do \
if echo $$tag | grep -q -e '-canary$$'; then \
: "creating or overwriting canary image"; \
push_image; \
elif docker pull $(IMAGE_NAME):$$tag 2>&1 | tee /dev/stderr | grep -q "manifest for $(IMAGE_NAME):$$tag not found"; then \
: "creating release image"; \
push_image; \
else \
: "release image $(IMAGE_NAME):$$tag already exists, skipping push"; \
fi; \
done
-
build: $(CMDS:%=build-%)
container: $(CMDS:%=container-%)
push: $(CMDS:%=push-%)
-
clean:
-rm -rf bin
-
test:
go test `go list ./... | grep -v 'vendor'` $(TESTARGS)
go vet `go list ./... | grep -v vendor`
files=$$(find . -name '*.go' | grep -v './vendor'); \
if [ $$(gofmt -d $$files | wc -l) -ne 0 ]; then \
echo "formatting errors:"; \
gofmt -d $$files; \
false; \
fi
- - - -
- -
- - - -
- - -
- - -
 -
- - - -
-
-
- -
-
- -
- - - - - - -
- - - You can’t perform that action at this time. -
- - - - - - - - - -
- - You signed in with another tab or window. Reload to refresh your session. - You signed out in another tab or window. Reload to refresh your session. -
- - - - - - -
- Press h to open a hovercard with more details. -
- -
- - - - +# Copyright 2019 The Kubernetes Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +.PHONY: build-% build container-% container push-% push clean test + +# A space-separated list of all commands in the repository, must be +# set in main Makefile of a repository. +# CMDS= + +# This is the default. It can be overridden in the main Makefile after +# including build.make. +REGISTRY_NAME=quay.io/k8scsi + +# Revision that gets built into each binary via the main.version +# string. Uses the `git describe` output based on the most recent +# version tag with a short revision suffix or, if nothing has been +# tagged yet, just the revision. +# +# Beware that tags may also be missing in shallow clones as done by +# some CI systems (like TravisCI, which pulls only 50 commits). +REV=$(shell git describe --long --tags --match='v*' --dirty 2>/dev/null || git rev-list -n1 HEAD) + +# A space-separated list of image tags under which the current build is to be pushed. +# Determined dynamically. +IMAGE_TAGS= + +# A "canary" image gets built if the current commit is the head of the remote "master" branch. +# That branch does not exist when building some other branch in TravisCI. +IMAGE_TAGS+=$(shell if [ "$$(git rev-list -n1 HEAD)" = "$$(git rev-list -n1 origin/master 2>/dev/null)" ]; then echo "canary"; fi) + +# A "X.Y.Z-canary" image gets built if the current commit is the head of a "origin/release-X.Y.Z" branch. +# The actual suffix does not matter, only the "release-" prefix is checked. +IMAGE_TAGS+=$(shell git branch -r --points-at=HEAD | grep 'origin/release-' | grep -v -e ' -> ' | sed -e 's;.*/release-\(.*\);\1-canary;') + +# A release image "vX.Y.Z" gets built if there is a tag of that format for the current commit. +# --abbrev=0 suppresses long format, only showing the closest tag. +IMAGE_TAGS+=$(shell tagged="$$(git describe --tags --match='v*' --abbrev=0)"; if [ "$$tagged" ] && [ "$$(git rev-list -n1 HEAD)" = "$$(git rev-list -n1 $$tagged)" ]; then echo $$tagged; fi) + +# Images are named after the command contained in them. +IMAGE_NAME=$(REGISTRY_NAME)/$* + +ifdef V +TESTARGS = -v -args -alsologtostderr -v 5 +else +TESTARGS = +endif + +build-%: + mkdir -p bin + CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$* ./cmd/$* + +container-%: build-% + docker build -t $*:latest -f $(shell if [ -e ./cmd/$*/Dockerfile ]; then echo ./cmd/$*/Dockerfile; else echo Dockerfile; fi) --label revision=$(REV) . + +push-%: container-% + set -ex; \ + push_image () { \ + docker tag $*:latest $(IMAGE_NAME):$$tag; \ + docker push $(IMAGE_NAME):$$tag; \ + }; \ + for tag in $(IMAGE_TAGS); do \ + if echo $$tag | grep -q -e '-canary$$'; then \ + : "creating or overwriting canary image"; \ + push_image; \ + elif docker pull $(IMAGE_NAME):$$tag 2>&1 | tee /dev/stderr | grep -q "manifest for $(IMAGE_NAME):$$tag not found"; then \ + : "creating release image"; \ + push_image; \ + else \ + : "release image $(IMAGE_NAME):$$tag already exists, skipping push"; \ + fi; \ + done + +build: $(CMDS:%=build-%) +container: $(CMDS:%=container-%) +push: $(CMDS:%=push-%) + +clean: + -rm -rf bin + +test: + go test `go list ./... | grep -v 'vendor'` $(TESTARGS) + go vet `go list ./... | grep -v vendor` + files=$$(find . -name '*.go' | grep -v './vendor'); \ + if [ $$(gofmt -d $$files | wc -l) -ne 0 ]; then \ + echo "formatting errors:"; \ + gofmt -d $$files; \ + false; \ + fi From 51e16be615359fa450d223f3a7381a85dacc62fb Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Mon, 21 Jan 2019 10:18:56 +0100 Subject: [PATCH 005/107] RELEASE.md: clarify release process It's worth calling out explicitly that only the master branch is maintained. --- RELEASE.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/RELEASE.md b/RELEASE.md index 491044d..a0fd815 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -1,3 +1,5 @@ # Release Process -TODO: describe the release process for this project +No tagged releases are planned at this point. The intention is to keep +the master branch in a state such that it can be used for all +supported branches in downstream repos which use these files. From c876547b7f4b22447c963a68cc81529e181105ba Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Mon, 21 Jan 2019 10:26:11 +0100 Subject: [PATCH 006/107] README.md: update repo name, add initial setup step The actual repository was not named like the prototype repo. --- README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index c67e930..286be88 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# [csi-build-rules](https://github.com/kubernetes-csi/csi-build-rules) +# [csi-release-tools](https://github.com/kubernetes-csi/csi-release-tools) These build and test rules can be shared between different Go projects without modifications. Customization for the different projects happen @@ -19,9 +19,9 @@ The expected repository layout is: - `cmd/*/*.go` - source code for each command - `cmd/*/Dockerfile` - docker file for each command or Dockerfile in the root when only building a single command - - `Makefile` - includes `build-rules/build.make` and sets + - `Makefile` - includes `release-tools/build.make` and sets configuration variables - - `.travis.yml` - a symlink to `build-rules/.travis.yml` + - `.travis.yml` - a symlink to `release-tools/.travis.yml` To create a release, tag a certain revision with a name that starts with `v`, for example `v1.0.0`, then `make push` @@ -40,11 +40,12 @@ Sharing and updating [`git subtree`](https://github.com/git/git/blob/master/contrib/subtree/git-subtree.txt) is the recommended way of maintaining a copy of the rules inside the -`build-rules` directory of a project. This way, it is possible to make +`release-tools` directory of a project. This way, it is possible to make changes also locally, test them and then push them back to the shared repository at a later time. Cheat sheet: -- `git subtree pull --prefix=build-rules https://github.com/kubernetes-csi/csi-build-rules.git master` - update local copy to latest upstream -- edit, `git commit`, `git subtree push --prefix=build-rules git@github.com:/csi-build-rules.git ` - push to a new branch before submitting a PR +- `git subtree add --prefix=release-tools https://github.com/pohly/csi-release-tools.git master` - add release tools to a repo which does not have them yet (only once) +- `git subtree pull --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master` - update local copy to latest upstream (whenever upstream changes) +- edit, `git commit`, `git subtree push --prefix=release-tools git@github.com:/csi-release-tools.git ` - push to a new branch before submitting a PR From f49080e0865b38254f20b372cf6da8aab23917b9 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Tue, 22 Jan 2019 10:28:11 +0100 Subject: [PATCH 007/107] README.md: fix repo URL for initial setup Copy-and-paste error from the time when the kubernetes-csi/csi-release-tools repo didn't have the code... --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 286be88..56d2248 100644 --- a/README.md +++ b/README.md @@ -46,6 +46,6 @@ repository at a later time. Cheat sheet: -- `git subtree add --prefix=release-tools https://github.com/pohly/csi-release-tools.git master` - add release tools to a repo which does not have them yet (only once) +- `git subtree add --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master` - add release tools to a repo which does not have them yet (only once) - `git subtree pull --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master` - update local copy to latest upstream (whenever upstream changes) - edit, `git commit`, `git subtree push --prefix=release-tools git@github.com:/csi-release-tools.git ` - push to a new branch before submitting a PR From 73db45967f57f911fbe707032c7b241af82ad5d7 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Wed, 23 Jan 2019 14:41:26 +0100 Subject: [PATCH 008/107] verify-subtree.sh: ensure that there are no local commits The goal is to enforce that changes get merged upstream first and only get into the local repo via a normal "git subtree merge". --- verify-subtree.sh | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100755 verify-subtree.sh diff --git a/verify-subtree.sh b/verify-subtree.sh new file mode 100755 index 0000000..ce8375f --- /dev/null +++ b/verify-subtree.sh @@ -0,0 +1,41 @@ +#! /bin/sh -e +# +# Copyright 2019 The Kubernetes Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This script verifies that the content of a directory managed +# by "git subtree" has not been modified locally. It does that +# by looking for commits that modify the files with the +# subtree prefix (aka directory) while ignoring merge +# commits. Merge commits are where "git subtree" pulls the +# upstream files into the directory. +# +# Theoretically a developer can subvert this check by modifying files +# in a merge commit, but in practice that shouldn't happen. + +DIR="$1" +if [ ! "$DIR" ]; then + echo "usage: $0 " >&2 + exit 1 +fi + +REV=$(git log -n1 --format=format:%H --no-merges -- "$DIR") +if [ "$REV" ]; then + echo "Directory '$DIR' contains non-upstream changes:" + echo + git log --no-merges -- "$DIR" + exit 1 +else + echo "$DIR is a clean copy of upstream." +fi From 9132a016e9465b0f6af506a9aa372b45aa1bfbe0 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Wed, 23 Jan 2019 14:49:07 +0100 Subject: [PATCH 009/107] test: split up into individual targets, run all "make test" used to abort after the first test failure. That was partly intentional: if the simple tests already fail (for example, because of a syntax error), then there is no point in continuing to test. However, it also makes it harder to find all errors in a CI system when the errors are unrelated (first error shows up, gets fixed, next error shows up, etc.). Now "make test" still aborts early, but "make -k test" is used in the CI and will run all individual tests because they are split up into different targets. --- build.make | 15 +++++++++++++++ travis.yml | 2 +- 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/build.make b/build.make index 4214648..4818b51 100644 --- a/build.make +++ b/build.make @@ -89,8 +89,23 @@ clean: -rm -rf bin test: + +.PHONY: test-go +test: test-go +test-go: + @ echo; echo $@ go test `go list ./... | grep -v 'vendor'` $(TESTARGS) + +.PHONY: test-vet +test: test-vet +test-vet: + @ echo; echo $@ go vet `go list ./... | grep -v vendor` + +.PHONY: test-fmt +test: test-fmt +test-fmt: + @ echo; echo $@ files=$$(find . -name '*.go' | grep -v './vendor'); \ if [ $$(gofmt -d $$files | wc -l) -ne 0 ]; then \ echo "formatting errors:"; \ diff --git a/travis.yml b/travis.yml index a44b001..b5a360a 100644 --- a/travis.yml +++ b/travis.yml @@ -6,7 +6,7 @@ matrix: include: - go: 1.11.1 script: -- make all test +- make -k all test after_success: - if [ "${TRAVIS_PULL_REQUEST}" == "false" ]; then docker login -u "${DOCKER_USERNAME}" -p "${DOCKER_PASSWORD}" quay.io; From f28c6b60aaa5f1ab61cb606744f25cc0ed8081a1 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Wed, 23 Jan 2019 14:52:09 +0100 Subject: [PATCH 010/107] test: verify that 'release-tools' subtree is clean We don't want to allow local modifications in the subtree. Everything should go to the csi-release-tools repo first. --- build.make | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/build.make b/build.make index 4818b51..f07f25e 100644 --- a/build.make +++ b/build.make @@ -112,3 +112,9 @@ test-fmt: gofmt -d $$files; \ false; \ fi + +.PHONY: test-subtree +test: test-subtree +test-subtree: + @ echo; echo $@ + ./release-tools/verify-subtree.sh release-tools From 154e33d434e49657651e669956eac56c7d43e073 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Thu, 24 Jan 2019 16:42:23 +0100 Subject: [PATCH 011/107] build.make: clarify usage of "make V=1" This may or may not work, depending on which packages have tests and whether they contain glog. --- build.make | 1 + 1 file changed, 1 insertion(+) diff --git a/build.make b/build.make index f07f25e..010dcb9 100644 --- a/build.make +++ b/build.make @@ -51,6 +51,7 @@ IMAGE_TAGS+=$(shell tagged="$$(git describe --tags --match='v*' --abbrev=0)"; if IMAGE_NAME=$(REGISTRY_NAME)/$* ifdef V +# Adding "-alsologtostderr" assumes that all test binaries contain glog. This is not guaranteed. TESTARGS = -v -args -alsologtostderr -v 5 else TESTARGS = From 147892c95429680cf2caa4a6ce32461fa50422d3 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Thu, 24 Jan 2019 16:43:24 +0100 Subject: [PATCH 012/107] build.make: support suppressing checks Individual repos may have to filter out certain packages from testing. For example, in csi-test the cmd/csi-sanity directory contains a special test that depends on additional parameters that set the CSI driver to test against. --- build.make | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/build.make b/build.make index 010dcb9..dfc405e 100644 --- a/build.make +++ b/build.make @@ -57,6 +57,9 @@ else TESTARGS = endif +# Specific packages can be excluded from each of the tests below by setting the *_FILTER_CMD variables +# to something like "| grep -v 'github.com/kubernetes-csi/project/pkg/foobar'". See usage below. + build-%: mkdir -p bin CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$* ./cmd/$* @@ -95,19 +98,19 @@ test: test: test-go test-go: @ echo; echo $@ - go test `go list ./... | grep -v 'vendor'` $(TESTARGS) + go test `go list ./... | grep -v 'vendor' $(TEST_GO_FILTER_CMD)` $(TESTARGS) .PHONY: test-vet test: test-vet test-vet: @ echo; echo $@ - go vet `go list ./... | grep -v vendor` + go vet `go list ./... | grep -v vendor $(TEST_VET_FILTER_CMD)` .PHONY: test-fmt test: test-fmt test-fmt: @ echo; echo $@ - files=$$(find . -name '*.go' | grep -v './vendor'); \ + files=$$(find . -name '*.go' | grep -v './vendor' $(TEST_FMT_FILTER_CMD)); \ if [ $$(gofmt -d $$files | wc -l) -ne 0 ]; then \ echo "formatting errors:"; \ gofmt -d $$files; \ From 09436b9f9011cbb997f9226f243347bd4e802a3d Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Fri, 25 Jan 2019 11:48:59 +0100 Subject: [PATCH 013/107] build.make: fix pushing of "canary" image from master branch After merging into external-attacher, the next Travis CI run did not push the "canary" image because the check for "canary" only covered the case where "-canary" is used as suffix (https://travis-ci.org/kubernetes-csi/external-attacher/builds/484095261). --- build.make | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.make b/build.make index f07f25e..a6a8916 100644 --- a/build.make +++ b/build.make @@ -70,7 +70,7 @@ push-%: container-% docker push $(IMAGE_NAME):$$tag; \ }; \ for tag in $(IMAGE_TAGS); do \ - if echo $$tag | grep -q -e '-canary$$'; then \ + if [ "$$tag" = "canary" ] || echo "$$tag" | grep -q -e '-canary$$'; then \ : "creating or overwriting canary image"; \ push_image; \ elif docker pull $(IMAGE_NAME):$$tag 2>&1 | tee /dev/stderr | grep -q "manifest for $(IMAGE_NAME):$$tag not found"; then \ From b0336b553c3770a34f67336a320ef8d1f77b28f8 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Fri, 25 Jan 2019 11:53:36 +0100 Subject: [PATCH 014/107] build.make: more readable "make test" output The introduction for each individual test looked like an actual command: test-subtree ./release-tools/verify-subtree.sh release-tools Directory 'release-tools' contains non-upstream changes: ... It's better to make it look like a shell comment and increase its visibility with a longer prefix: ### test-subtree: ./release-tools/verify-subtree.sh release-tools ... --- build.make | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/build.make b/build.make index dfc405e..4296ff4 100644 --- a/build.make +++ b/build.make @@ -97,19 +97,19 @@ test: .PHONY: test-go test: test-go test-go: - @ echo; echo $@ + @ echo; echo "### $@:" go test `go list ./... | grep -v 'vendor' $(TEST_GO_FILTER_CMD)` $(TESTARGS) .PHONY: test-vet test: test-vet test-vet: - @ echo; echo $@ + @ echo; echo "### $@:" go vet `go list ./... | grep -v vendor $(TEST_VET_FILTER_CMD)` .PHONY: test-fmt test: test-fmt test-fmt: - @ echo; echo $@ + @ echo; echo "### $@:" files=$$(find . -name '*.go' | grep -v './vendor' $(TEST_FMT_FILTER_CMD)); \ if [ $$(gofmt -d $$files | wc -l) -ne 0 ]; then \ echo "formatting errors:"; \ @@ -120,5 +120,5 @@ test-fmt: .PHONY: test-subtree test: test-subtree test-subtree: - @ echo; echo $@ + @ echo; echo "### $@:" ./release-tools/verify-subtree.sh release-tools From cc564f929a4e32a829d35272533242b94797aa22 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Wed, 6 Mar 2019 17:23:07 +0100 Subject: [PATCH 015/107] verify-subtree.sh: relax check and ignore old content If for whatever reasons a repo already had a `release-tools` directory before doing a clean import of it with `git subtree`, the check used to fail because it found those old commits. This can be fixed by telling `git log` to stop when the directory disappears from the repo. There has to be a commit with removes the old content, because otherwise `git subtree add` doesn't work. Fixes: https://github.com/kubernetes-csi/external-resizer/issues/21 --- verify-subtree.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/verify-subtree.sh b/verify-subtree.sh index ce8375f..f04a9fa 100755 --- a/verify-subtree.sh +++ b/verify-subtree.sh @@ -30,7 +30,7 @@ if [ ! "$DIR" ]; then exit 1 fi -REV=$(git log -n1 --format=format:%H --no-merges -- "$DIR") +REV=$(git log -n1 --remove-empty --format=format:%H --no-merges -- "$DIR") if [ "$REV" ]; then echo "Directory '$DIR' contains non-upstream changes:" echo From 94fc1e31d8acc93663992d117ecf9fc3ca9fceb0 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Fri, 15 Mar 2019 11:08:53 +0100 Subject: [PATCH 016/107] build.make: avoid unit-testing E2E test suite In repos that have a test/e2e, that test suite should be run separately because it depends on a running cluster. --- build.make | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.make b/build.make index 8ca0b2c..6ea3279 100644 --- a/build.make +++ b/build.make @@ -98,7 +98,7 @@ test: test: test-go test-go: @ echo; echo "### $@:" - go test `go list ./... | grep -v 'vendor' $(TEST_GO_FILTER_CMD)` $(TESTARGS) + go test `go list ./... | grep -v -e 'vendor' -e '/test/e2e$$' $(TEST_GO_FILTER_CMD)` $(TESTARGS) .PHONY: test-vet test: test-vet From fb13c5198f899657baac7bb44156abc6d0f20724 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Fri, 15 Mar 2019 11:55:06 +0100 Subject: [PATCH 017/107] verify-shellcheck.sh: import from Kubernetes This is an unmodified copy of kubernetes/hack/verify-shellcheck.sh revision d5a3db003916b1d33b503ccd2e4897e094d8af77. --- util.sh | 839 +++++++++++++++++++++++++++++++++++++++++++ verify-shellcheck.sh | 187 ++++++++++ 2 files changed, 1026 insertions(+) create mode 100755 util.sh create mode 100755 verify-shellcheck.sh diff --git a/util.sh b/util.sh new file mode 100755 index 0000000..31ce9fc --- /dev/null +++ b/util.sh @@ -0,0 +1,839 @@ +#!/usr/bin/env bash + +# Copyright 2014 The Kubernetes Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +function kube::util::sourced_variable { + # Call this function to tell shellcheck that a variable is supposed to + # be used from other calling context. This helps quiet an "unused + # variable" warning from shellcheck and also document your code. + true +} + +kube::util::sortable_date() { + date "+%Y%m%d-%H%M%S" +} + +# arguments: target, item1, item2, item3, ... +# returns 0 if target is in the given items, 1 otherwise. +kube::util::array_contains() { + local search="$1" + local element + shift + for element; do + if [[ "${element}" == "${search}" ]]; then + return 0 + fi + done + return 1 +} + +kube::util::wait_for_url() { + local url=$1 + local prefix=${2:-} + local wait=${3:-1} + local times=${4:-30} + local maxtime=${5:-1} + + command -v curl >/dev/null || { + kube::log::usage "curl must be installed" + exit 1 + } + + local i + for i in $(seq 1 "${times}"); do + local out + if out=$(curl --max-time "${maxtime}" -gkfs "${url}" 2>/dev/null); then + kube::log::status "On try ${i}, ${prefix}: ${out}" + return 0 + fi + sleep "${wait}" + done + kube::log::error "Timed out waiting for ${prefix} to answer at ${url}; tried ${times} waiting ${wait} between each" + return 1 +} + +# Example: kube::util::trap_add 'echo "in trap DEBUG"' DEBUG +# See: http://stackoverflow.com/questions/3338030/multiple-bash-traps-for-the-same-signal +kube::util::trap_add() { + local trap_add_cmd + trap_add_cmd=$1 + shift + + for trap_add_name in "$@"; do + local existing_cmd + local new_cmd + + # Grab the currently defined trap commands for this trap + existing_cmd=$(trap -p "${trap_add_name}" | awk -F"'" '{print $2}') + + if [[ -z "${existing_cmd}" ]]; then + new_cmd="${trap_add_cmd}" + else + new_cmd="${trap_add_cmd};${existing_cmd}" + fi + + # Assign the test. Disable the shellcheck warning telling that trap + # commands should be single quoted to avoid evaluating them at this + # point instead evaluating them at run time. The logic of adding new + # commands to a single trap requires them to be evaluated right away. + # shellcheck disable=SC2064 + trap "${new_cmd}" "${trap_add_name}" + done +} + +# Opposite of kube::util::ensure-temp-dir() +kube::util::cleanup-temp-dir() { + rm -rf "${KUBE_TEMP}" +} + +# Create a temp dir that'll be deleted at the end of this bash session. +# +# Vars set: +# KUBE_TEMP +kube::util::ensure-temp-dir() { + if [[ -z ${KUBE_TEMP-} ]]; then + KUBE_TEMP=$(mktemp -d 2>/dev/null || mktemp -d -t kubernetes.XXXXXX) + kube::util::trap_add kube::util::cleanup-temp-dir EXIT + fi +} + +kube::util::host_os() { + local host_os + case "$(uname -s)" in + Darwin) + host_os=darwin + ;; + Linux) + host_os=linux + ;; + *) + kube::log::error "Unsupported host OS. Must be Linux or Mac OS X." + exit 1 + ;; + esac + echo "${host_os}" +} + +kube::util::host_arch() { + local host_arch + case "$(uname -m)" in + x86_64*) + host_arch=amd64 + ;; + i?86_64*) + host_arch=amd64 + ;; + amd64*) + host_arch=amd64 + ;; + aarch64*) + host_arch=arm64 + ;; + arm64*) + host_arch=arm64 + ;; + arm*) + host_arch=arm + ;; + i?86*) + host_arch=x86 + ;; + s390x*) + host_arch=s390x + ;; + ppc64le*) + host_arch=ppc64le + ;; + *) + kube::log::error "Unsupported host arch. Must be x86_64, 386, arm, arm64, s390x or ppc64le." + exit 1 + ;; + esac + echo "${host_arch}" +} + +# This figures out the host platform without relying on golang. We need this as +# we don't want a golang install to be a prerequisite to building yet we need +# this info to figure out where the final binaries are placed. +kube::util::host_platform() { + echo "$(kube::util::host_os)/$(kube::util::host_arch)" +} + +kube::util::find-binary-for-platform() { + local -r lookfor="$1" + local -r platform="$2" + local locations=( + "${KUBE_ROOT}/_output/bin/${lookfor}" + "${KUBE_ROOT}/_output/dockerized/bin/${platform}/${lookfor}" + "${KUBE_ROOT}/_output/local/bin/${platform}/${lookfor}" + "${KUBE_ROOT}/platforms/${platform}/${lookfor}" + ) + # Also search for binary in bazel build tree. + # The bazel go rules place some binaries in subtrees like + # "bazel-bin/source/path/linux_amd64_pure_stripped/binaryname", so make sure + # the platform name is matched in the path. + while IFS=$'\n' read -r location; do + locations+=("$location"); + done < <(find "${KUBE_ROOT}/bazel-bin/" -type f -executable \ + \( -path "*/${platform/\//_}*/${lookfor}" -o -path "*/${lookfor}" \) 2>/dev/null || true) + + # List most recently-updated location. + local -r bin=$( (ls -t "${locations[@]}" 2>/dev/null || true) | head -1 ) + echo -n "${bin}" +} + +kube::util::find-binary() { + kube::util::find-binary-for-platform "$1" "$(kube::util::host_platform)" +} + +# Run all known doc generators (today gendocs and genman for kubectl) +# $1 is the directory to put those generated documents +kube::util::gen-docs() { + local dest="$1" + + # Find binary + gendocs=$(kube::util::find-binary "gendocs") + genkubedocs=$(kube::util::find-binary "genkubedocs") + genman=$(kube::util::find-binary "genman") + genyaml=$(kube::util::find-binary "genyaml") + genfeddocs=$(kube::util::find-binary "genfeddocs") + + # TODO: If ${genfeddocs} is not used from anywhere (it isn't used at + # least from k/k tree), remove it completely. + kube::util::sourced_variable "${genfeddocs}" + + mkdir -p "${dest}/docs/user-guide/kubectl/" + "${gendocs}" "${dest}/docs/user-guide/kubectl/" + mkdir -p "${dest}/docs/admin/" + "${genkubedocs}" "${dest}/docs/admin/" "kube-apiserver" + "${genkubedocs}" "${dest}/docs/admin/" "kube-controller-manager" + "${genkubedocs}" "${dest}/docs/admin/" "cloud-controller-manager" + "${genkubedocs}" "${dest}/docs/admin/" "kube-proxy" + "${genkubedocs}" "${dest}/docs/admin/" "kube-scheduler" + "${genkubedocs}" "${dest}/docs/admin/" "kubelet" + "${genkubedocs}" "${dest}/docs/admin/" "kubeadm" + + mkdir -p "${dest}/docs/man/man1/" + "${genman}" "${dest}/docs/man/man1/" "kube-apiserver" + "${genman}" "${dest}/docs/man/man1/" "kube-controller-manager" + "${genman}" "${dest}/docs/man/man1/" "cloud-controller-manager" + "${genman}" "${dest}/docs/man/man1/" "kube-proxy" + "${genman}" "${dest}/docs/man/man1/" "kube-scheduler" + "${genman}" "${dest}/docs/man/man1/" "kubelet" + "${genman}" "${dest}/docs/man/man1/" "kubectl" + "${genman}" "${dest}/docs/man/man1/" "kubeadm" + + mkdir -p "${dest}/docs/yaml/kubectl/" + "${genyaml}" "${dest}/docs/yaml/kubectl/" + + # create the list of generated files + pushd "${dest}" > /dev/null || return 1 + touch docs/.generated_docs + find . -type f | cut -sd / -f 2- | LC_ALL=C sort > docs/.generated_docs + popd > /dev/null || return 1 +} + +# Removes previously generated docs-- we don't want to check them in. $KUBE_ROOT +# must be set. +kube::util::remove-gen-docs() { + if [ -e "${KUBE_ROOT}/docs/.generated_docs" ]; then + # remove all of the old docs; we don't want to check them in. + while read -r file; do + rm "${KUBE_ROOT}/${file}" 2>/dev/null || true + done <"${KUBE_ROOT}/docs/.generated_docs" + # The docs/.generated_docs file lists itself, so we don't need to explicitly + # delete it. + fi +} + +# Takes a group/version and returns the path to its location on disk, sans +# "pkg". E.g.: +# * default behavior: extensions/v1beta1 -> apis/extensions/v1beta1 +# * default behavior for only a group: experimental -> apis/experimental +# * Special handling for empty group: v1 -> api/v1, unversioned -> api/unversioned +# * Special handling for groups suffixed with ".k8s.io": foo.k8s.io/v1 -> apis/foo/v1 +# * Very special handling for when both group and version are "": / -> api +kube::util::group-version-to-pkg-path() { + local group_version="$1" + + while IFS=$'\n' read -r api; do + if [[ "${api}" = "${group_version/.*k8s.io/}" ]]; then + echo "vendor/k8s.io/api/${group_version/.*k8s.io/}" + return + fi + done < <(cd "${KUBE_ROOT}/staging/src/k8s.io/api" && find . -name types.go -exec dirname {} \; | sed "s|\./||g" | sort) + + # "v1" is the API GroupVersion + if [[ "${group_version}" == "v1" ]]; then + echo "vendor/k8s.io/api/core/v1" + return + fi + + # Special cases first. + # TODO(lavalamp): Simplify this by moving pkg/api/v1 and splitting pkg/api, + # moving the results to pkg/apis/api. + case "${group_version}" in + # both group and version are "", this occurs when we generate deep copies for internal objects of the legacy v1 API. + __internal) + echo "pkg/apis/core" + ;; + meta/v1) + echo "vendor/k8s.io/apimachinery/pkg/apis/meta/v1" + ;; + meta/v1beta1) + echo "vendor/k8s.io/apimachinery/pkg/apis/meta/v1beta1" + ;; + *.k8s.io) + echo "pkg/apis/${group_version%.*k8s.io}" + ;; + *.k8s.io/*) + echo "pkg/apis/${group_version/.*k8s.io/}" + ;; + *) + echo "pkg/apis/${group_version%__internal}" + ;; + esac +} + +# Takes a group/version and returns the swagger-spec file name. +# default behavior: extensions/v1beta1 -> extensions_v1beta1 +# special case for v1: v1 -> v1 +kube::util::gv-to-swagger-name() { + local group_version="$1" + case "${group_version}" in + v1) + echo "v1" + ;; + *) + echo "${group_version%/*}_${group_version#*/}" + ;; + esac +} + +# Returns the name of the upstream remote repository name for the local git +# repo, e.g. "upstream" or "origin". +kube::util::git_upstream_remote_name() { + git remote -v | grep fetch |\ + grep -E 'github.com[/:]kubernetes/kubernetes|k8s.io/kubernetes' |\ + head -n 1 | awk '{print $1}' +} + +# Ensures the current directory is a git tree for doing things like restoring or +# validating godeps +kube::util::create-fake-git-tree() { + local -r target_dir=${1:-$(pwd)} + + pushd "${target_dir}" >/dev/null || return 1 + git init >/dev/null + git config --local user.email "nobody@k8s.io" + git config --local user.name "$0" + git add . >/dev/null + git commit -q -m "Snapshot" >/dev/null + if (( ${KUBE_VERBOSE:-5} >= 6 )); then + kube::log::status "${target_dir} is now a git tree." + fi + popd >/dev/null || return 1 +} + +# Checks whether godep restore was run in the current GOPATH, i.e. that all referenced repos exist +# and are checked out to the referenced rev. +kube::util::godep_restored() { + local -r godeps_json=${1:-Godeps/Godeps.json} + local -r gopath=${2:-${GOPATH%:*}} + + kube::util::require-jq + + local root + local old_rev="" + while read -r path rev; do + rev="${rev//[\'\"]}" # remove quotes which are around revs sometimes + + if [[ "${rev}" == "${old_rev}" ]] && [[ "${path}" == "${root}"* ]]; then + # avoid checking the same git/hg root again + continue + fi + + root="${path}" + while [ "${root}" != "." ] && [ ! -d "${gopath}/src/${root}/.git" ] && [ ! -d "${gopath}/src/${root}/.hg" ]; do + root=$(dirname "${root}") + done + if [ "${root}" == "." ]; then + echo "No checkout of ${path} found in GOPATH \"${gopath}\"." 1>&2 + return 1 + fi + local head + if [ -d "${gopath}/src/${root}/.git" ]; then + head="$(cd "${gopath}/src/${root}" && git rev-parse HEAD)" + else + head="$(cd "${gopath}/src/${root}" && hg parent --template '{node}')" + fi + if [ "${head}" != "${rev}" ]; then + echo "Unexpected HEAD '${head}' at ${gopath}/src/${root}, expected '${rev}'." 1>&2 + return 1 + fi + old_rev="${rev}" + done < <(jq '.Deps|.[]|.ImportPath + " " + .Rev' -r < "${godeps_json}") + return 0 +} + +# Exits script if working directory is dirty. If it's run interactively in the terminal +# the user can commit changes in a second terminal. This script will wait. +kube::util::ensure_clean_working_dir() { + while ! git diff HEAD --exit-code &>/dev/null; do + echo -e "\nUnexpected dirty working directory:\n" + if tty -s; then + git status -s + else + git diff -a # be more verbose in log files without tty + exit 1 + fi | sed 's/^/ /' + echo -e "\nCommit your changes in another terminal and then continue here by pressing enter." + read -r + done 1>&2 +} + +# Ensure that the given godep version is installed and in the path. Almost +# nobody should use any version but the default. +# +# Sets: +# KUBE_GODEP: The path to the godep binary +# +kube::util::ensure_godep_version() { + local godep_target_version=${1:-"v80-k8s-r1"} # this version is known to work + + # If KUBE_GODEP is already set, and it's the right version, then use it. + if [[ -n "${KUBE_GODEP:-}" && "$(${KUBE_GODEP:?} version 2>/dev/null)" == *"godep ${godep_target_version}"* ]]; then + kube::log::status "Using ${KUBE_GODEP}" + return + fi + + # Otherwise, install forked godep + kube::log::status "Installing godep version ${godep_target_version}" + GOBIN="${KUBE_OUTPUT_BINPATH}" go install k8s.io/kubernetes/third_party/forked/godep + export KUBE_GODEP="${KUBE_OUTPUT_BINPATH}/godep" + kube::log::status "Installed ${KUBE_GODEP}" + + # Verify that the installed godep from fork is what we expect + if [[ "$(${KUBE_GODEP:?} version 2>/dev/null)" != *"godep ${godep_target_version}"* ]]; then + kube::log::error "Expected godep ${godep_target_version} from ${KUBE_GODEP}, got $(${KUBE_GODEP:?} version)" + return 1 + fi +} + +# Ensure that none of the staging repos is checked out in the GOPATH because this +# easily confused godep. +kube::util::ensure_no_staging_repos_in_gopath() { + kube::util::ensure_single_dir_gopath + local error=0 + for repo_file in "${KUBE_ROOT}"/staging/src/k8s.io/*; do + if [[ ! -d "${repo_file}" ]]; then + # not a directory or there were no files + continue; + fi + repo="$(basename "${repo_file}")" + if [ -e "${GOPATH}/src/k8s.io/${repo}" ]; then + echo "k8s.io/${repo} exists in GOPATH. Remove before running godep-save.sh." 1>&2 + error=1 + fi + done + if [ "${error}" = "1" ]; then + exit 1 + fi +} + +# Checks that the GOPATH is simple, i.e. consists only of one directory, not multiple. +kube::util::ensure_single_dir_gopath() { + if [[ "${GOPATH}" == *:* ]]; then + echo "GOPATH must consist of a single directory." 1>&2 + exit 1 + fi +} + +# Find the base commit using: +# $PULL_BASE_SHA if set (from Prow) +# current ref from the remote upstream branch +kube::util::base_ref() { + local -r git_branch=$1 + + if [[ -n ${PULL_BASE_SHA:-} ]]; then + echo "${PULL_BASE_SHA}" + return + fi + + full_branch="$(kube::util::git_upstream_remote_name)/${git_branch}" + + # make sure the branch is valid, otherwise the check will pass erroneously. + if ! git describe "${full_branch}" >/dev/null; then + # abort! + exit 1 + fi + + echo "${full_branch}" +} + +# Checks whether there are any files matching pattern $2 changed between the +# current branch and upstream branch named by $1. +# Returns 1 (false) if there are no changes +# 0 (true) if there are changes detected. +kube::util::has_changes() { + local -r git_branch=$1 + local -r pattern=$2 + local -r not_pattern=${3:-totallyimpossiblepattern} + + local base_ref + base_ref=$(kube::util::base_ref "${git_branch}") + echo "Checking for '${pattern}' changes against '${base_ref}'" + + # notice this uses ... to find the first shared ancestor + if git diff --name-only "${base_ref}...HEAD" | grep -v -E "${not_pattern}" | grep "${pattern}" > /dev/null; then + return 0 + fi + # also check for pending changes + if git status --porcelain | grep -v -E "${not_pattern}" | grep "${pattern}" > /dev/null; then + echo "Detected '${pattern}' uncommitted changes." + return 0 + fi + echo "No '${pattern}' changes detected." + return 1 +} + +kube::util::download_file() { + local -r url=$1 + local -r destination_file=$2 + + rm "${destination_file}" 2&> /dev/null || true + + for i in $(seq 5) + do + if ! curl -fsSL --retry 3 --keepalive-time 2 "${url}" -o "${destination_file}"; then + echo "Downloading ${url} failed. $((5-i)) retries left." + sleep 1 + else + echo "Downloading ${url} succeed" + return 0 + fi + done + return 1 +} + +# Test whether openssl is installed. +# Sets: +# OPENSSL_BIN: The path to the openssl binary to use +function kube::util::test_openssl_installed { + if ! openssl version >& /dev/null; then + echo "Failed to run openssl. Please ensure openssl is installed" + exit 1 + fi + + OPENSSL_BIN=$(command -v openssl) +} + +# creates a client CA, args are sudo, dest-dir, ca-id, purpose +# purpose is dropped in after "key encipherment", you usually want +# '"client auth"' +# '"server auth"' +# '"client auth","server auth"' +function kube::util::create_signing_certkey { + local sudo=$1 + local dest_dir=$2 + local id=$3 + local purpose=$4 + # Create client ca + ${sudo} /usr/bin/env bash -e < "${dest_dir}/${id}-ca-config.json" +EOF +} + +# signs a client certificate: args are sudo, dest-dir, CA, filename (roughly), username, groups... +function kube::util::create_client_certkey { + local sudo=$1 + local dest_dir=$2 + local ca=$3 + local id=$4 + local cn=${5:-$4} + local groups="" + local SEP="" + shift 5 + while [ -n "${1:-}" ]; do + groups+="${SEP}{\"O\":\"$1\"}" + SEP="," + shift 1 + done + ${sudo} /usr/bin/env bash -e < /dev/null +apiVersion: v1 +kind: Config +clusters: + - cluster: + certificate-authority: ${ca_file} + server: https://${api_host}:${api_port}/ + name: local-up-cluster +users: + - user: + token: ${token} + client-certificate: ${dest_dir}/client-${client_id}.crt + client-key: ${dest_dir}/client-${client_id}.key + name: local-up-cluster +contexts: + - context: + cluster: local-up-cluster + user: local-up-cluster + name: local-up-cluster +current-context: local-up-cluster +EOF + + # flatten the kubeconfig files to make them self contained + username=$(whoami) + ${sudo} /usr/bin/env bash -e < "/tmp/${client_id}.kubeconfig" + mv -f "/tmp/${client_id}.kubeconfig" "${dest_dir}/${client_id}.kubeconfig" + chown ${username} "${dest_dir}/${client_id}.kubeconfig" +EOF +} + +# Determines if docker can be run, failures may simply require that the user be added to the docker group. +function kube::util::ensure_docker_daemon_connectivity { + IFS=" " read -ra DOCKER <<< "${DOCKER_OPTS}" + # Expand ${DOCKER[@]} only if it's not unset. This is to work around + # Bash 3 issue with unbound variable. + DOCKER=(docker ${DOCKER[@]:+"${DOCKER[@]}"}) + if ! "${DOCKER[@]}" info > /dev/null 2>&1 ; then + cat <<'EOF' >&2 +Can't connect to 'docker' daemon. please fix and retry. + +Possible causes: + - Docker Daemon not started + - Linux: confirm via your init system + - macOS w/ docker-machine: run `docker-machine ls` and `docker-machine start ` + - macOS w/ Docker for Mac: Check the menu bar and start the Docker application + - DOCKER_HOST hasn't been set or is set incorrectly + - Linux: domain socket is used, DOCKER_* should be unset. In Bash run `unset ${!DOCKER_*}` + - macOS w/ docker-machine: run `eval "$(docker-machine env )"` + - macOS w/ Docker for Mac: domain socket is used, DOCKER_* should be unset. In Bash run `unset ${!DOCKER_*}` + - Other things to check: + - Linux: User isn't in 'docker' group. Add and relogin. + - Something like 'sudo usermod -a -G docker ${USER}' + - RHEL7 bug and workaround: https://bugzilla.redhat.com/show_bug.cgi?id=1119282#c8 +EOF + return 1 + fi +} + +# Wait for background jobs to finish. Return with +# an error status if any of the jobs failed. +kube::util::wait-for-jobs() { + local fail=0 + local job + for job in $(jobs -p); do + wait "${job}" || fail=$((fail + 1)) + done + return ${fail} +} + +# kube::util::join +# Concatenates the list elements with the delimiter passed as first parameter +# +# Ex: kube::util::join , a b c +# -> a,b,c +function kube::util::join { + local IFS="$1" + shift + echo "$*" +} + +# Downloads cfssl/cfssljson into $1 directory if they do not already exist in PATH +# +# Assumed vars: +# $1 (cfssl directory) (optional) +# +# Sets: +# CFSSL_BIN: The path of the installed cfssl binary +# CFSSLJSON_BIN: The path of the installed cfssljson binary +# +function kube::util::ensure-cfssl { + if command -v cfssl &>/dev/null && command -v cfssljson &>/dev/null; then + CFSSL_BIN=$(command -v cfssl) + CFSSLJSON_BIN=$(command -v cfssljson) + return 0 + fi + + host_arch=$(kube::util::host_arch) + + if [[ "${host_arch}" != "amd64" ]]; then + echo "Cannot download cfssl on non-amd64 hosts and cfssl does not appear to be installed." + echo "Please install cfssl and cfssljson and verify they are in \$PATH." + echo "Hint: export PATH=\$PATH:\$GOPATH/bin; go get -u github.com/cloudflare/cfssl/cmd/..." + exit 1 + fi + + # Create a temp dir for cfssl if no directory was given + local cfssldir=${1:-} + if [[ -z "${cfssldir}" ]]; then + kube::util::ensure-temp-dir + cfssldir="${KUBE_TEMP}/cfssl" + fi + + mkdir -p "${cfssldir}" + pushd "${cfssldir}" > /dev/null || return 1 + + echo "Unable to successfully run 'cfssl' from ${PATH}; downloading instead..." + kernel=$(uname -s) + case "${kernel}" in + Linux) + curl --retry 10 -L -o cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 + curl --retry 10 -L -o cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 + ;; + Darwin) + curl --retry 10 -L -o cfssl https://pkg.cfssl.org/R1.2/cfssl_darwin-amd64 + curl --retry 10 -L -o cfssljson https://pkg.cfssl.org/R1.2/cfssljson_darwin-amd64 + ;; + *) + echo "Unknown, unsupported platform: ${kernel}." >&2 + echo "Supported platforms: Linux, Darwin." >&2 + exit 2 + esac + + chmod +x cfssl || true + chmod +x cfssljson || true + + CFSSL_BIN="${cfssldir}/cfssl" + CFSSLJSON_BIN="${cfssldir}/cfssljson" + if [[ ! -x ${CFSSL_BIN} || ! -x ${CFSSLJSON_BIN} ]]; then + echo "Failed to download 'cfssl'. Please install cfssl and cfssljson and verify they are in \$PATH." + echo "Hint: export PATH=\$PATH:\$GOPATH/bin; go get -u github.com/cloudflare/cfssl/cmd/..." + exit 1 + fi + popd > /dev/null || return 1 +} + +# kube::util::ensure_dockerized +# Confirms that the script is being run inside a kube-build image +# +function kube::util::ensure_dockerized { + if [[ -f /kube-build-image ]]; then + return 0 + else + echo "ERROR: This script is designed to be run inside a kube-build container" + exit 1 + fi +} + +# kube::util::ensure-gnu-sed +# Determines which sed binary is gnu-sed on linux/darwin +# +# Sets: +# SED: The name of the gnu-sed binary +# +function kube::util::ensure-gnu-sed { + if LANG=C sed --help 2>&1 | grep -q GNU; then + SED="sed" + elif command -v gsed &>/dev/null; then + SED="gsed" + else + kube::log::error "Failed to find GNU sed as sed or gsed. If you are on Mac: brew install gnu-sed." >&2 + return 1 + fi + kube::util::sourced_variable "${SED}" +} + +# kube::util::check-file-in-alphabetical-order +# Check that the file is in alphabetical order +# +function kube::util::check-file-in-alphabetical-order { + local failure_file="$1" + if ! diff -u "${failure_file}" <(LC_ALL=C sort "${failure_file}"); then + { + echo + echo "${failure_file} is not in alphabetical order. Please sort it:" + echo + echo " LC_ALL=C sort -o ${failure_file} ${failure_file}" + echo + } >&2 + false + fi +} + +# kube::util::require-jq +# Checks whether jq is installed. +function kube::util::require-jq { + if ! command -v jq &>/dev/null; then + echo "jq not found. Please install." 1>&2 + return 1 + fi +} + +# Some useful colors. +if [[ -z "${color_start-}" ]]; then + declare -r color_start="\033[" + declare -r color_red="${color_start}0;31m" + declare -r color_yellow="${color_start}0;33m" + declare -r color_green="${color_start}0;32m" + declare -r color_blue="${color_start}1;34m" + declare -r color_cyan="${color_start}1;36m" + declare -r color_norm="${color_start}0m" + + kube::util::sourced_variable "${color_start}" + kube::util::sourced_variable "${color_red}" + kube::util::sourced_variable "${color_yellow}" + kube::util::sourced_variable "${color_green}" + kube::util::sourced_variable "${color_blue}" + kube::util::sourced_variable "${color_cyan}" + kube::util::sourced_variable "${color_norm}" +fi + +# ex: ts=2 sw=2 et filetype=sh diff --git a/verify-shellcheck.sh b/verify-shellcheck.sh new file mode 100755 index 0000000..1b882fa --- /dev/null +++ b/verify-shellcheck.sh @@ -0,0 +1,187 @@ +#!/usr/bin/env bash + +# Copyright 2018 The Kubernetes Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -o errexit +set -o nounset +set -o pipefail + +KUBE_ROOT=$(dirname "${BASH_SOURCE[0]}")/.. +source "${KUBE_ROOT}/hack/lib/init.sh" +source "${KUBE_ROOT}/hack/lib/util.sh" + +# required version for this script, if not installed on the host we will +# use the official docker image instead. keep this in sync with SHELLCHECK_IMAGE +SHELLCHECK_VERSION="0.6.0" +# upstream shellcheck latest stable image as of January 10th, 2019 +SHELLCHECK_IMAGE="koalaman/shellcheck-alpine:v0.6.0@sha256:7d4d712a2686da99d37580b4e2f45eb658b74e4b01caf67c1099adc294b96b52" + +# fixed name for the shellcheck docker container so we can reliably clean it up +SHELLCHECK_CONTAINER="k8s-shellcheck" + +# disabled lints +disabled=( + # this lint disallows non-constant source, which we use extensively without + # any known bugs + 1090 + # this lint prefers command -v to which, they are not the same + 2230 +) +# comma separate for passing to shellcheck +join_by() { + local IFS="$1"; + shift; + echo "$*"; +} +SHELLCHECK_DISABLED="$(join_by , "${disabled[@]}")" +readonly SHELLCHECK_DISABLED + +# creates the shellcheck container for later use +create_container () { + # TODO(bentheelder): this is a performance hack, we create the container with + # a sleep MAX_INT32 so that it is effectively paused. + # We then repeatedly exec to it to run each shellcheck, and later rm it when + # we're done. + # This is incredibly much faster than creating a container for each shellcheck + # call ... + docker run --name "${SHELLCHECK_CONTAINER}" -d --rm -v "${KUBE_ROOT}:${KUBE_ROOT}" -w "${KUBE_ROOT}" --entrypoint="sleep" "${SHELLCHECK_IMAGE}" 2147483647 +} +# removes the shellcheck container +remove_container () { + docker rm -f "${SHELLCHECK_CONTAINER}" &> /dev/null || true +} + +# ensure we're linting the k8s source tree +cd "${KUBE_ROOT}" + +# find all shell scripts excluding ./_*, ./.git/*, ./vendor*, +# and anything git-ignored +all_shell_scripts=() +while IFS=$'\n' read -r script; + do git check-ignore -q "$script" || all_shell_scripts+=("$script"); +done < <(find . -name "*.sh" \ + -not \( \ + -path ./_\* -o \ + -path ./.git\* -o \ + -path ./vendor\* \ + \)) + +# make sure known failures are sorted +failure_file="${KUBE_ROOT}/hack/.shellcheck_failures" +kube::util::check-file-in-alphabetical-order "${failure_file}" + +# load known failure files +failing_files=() +while IFS=$'\n' read -r script; + do failing_files+=("$script"); +done < <(cat "${failure_file}") + +# detect if the host machine has the required shellcheck version installed +# if so, we will use that instead. +HAVE_SHELLCHECK=false +if which shellcheck &>/dev/null; then + detected_version="$(shellcheck --version | grep 'version: .*')" + if [[ "${detected_version}" = "version: ${SHELLCHECK_VERSION}" ]]; then + HAVE_SHELLCHECK=true + fi +fi + +# tell the user which we've selected and possibly set up the container +if ${HAVE_SHELLCHECK}; then + echo "Using host shellcheck ${SHELLCHECK_VERSION} binary." +else + echo "Using shellcheck ${SHELLCHECK_VERSION} docker image." + # remove any previous container, ensure we will attempt to cleanup on exit, + # and create the container + remove_container + kube::util::trap_add 'remove_container' EXIT + if ! output="$(create_container 2>&1)"; then + { + echo "Failed to create shellcheck container with output: " + echo "" + echo "${output}" + } >&2 + exit 1 + fi +fi + +# lint each script, tracking failures +errors=() +not_failing=() +for f in "${all_shell_scripts[@]}"; do + set +o errexit + if ${HAVE_SHELLCHECK}; then + failedLint=$(shellcheck --exclude="${SHELLCHECK_DISABLED}" "${f}") + else + failedLint=$(docker exec -t ${SHELLCHECK_CONTAINER} \ + shellcheck --exclude="${SHELLCHECK_DISABLED}" "${f}") + fi + set -o errexit + kube::util::array_contains "${f}" "${failing_files[@]}" && in_failing=$? || in_failing=$? + if [[ -n "${failedLint}" ]] && [[ "${in_failing}" -ne "0" ]]; then + errors+=( "${failedLint}" ) + fi + if [[ -z "${failedLint}" ]] && [[ "${in_failing}" -eq "0" ]]; then + not_failing+=( "${f}" ) + fi +done + +# Check to be sure all the packages that should pass lint are. +if [ ${#errors[@]} -eq 0 ]; then + echo 'Congratulations! All shell files are passing lint (excluding those in hack/.shellcheck_failures).' +else + { + echo "Errors from shellcheck:" + for err in "${errors[@]}"; do + echo "$err" + done + echo + echo 'Please review the above warnings. You can test via "./hack/verify-shellcheck"' + echo 'If the above warnings do not make sense, you can exempt this package from shellcheck' + echo 'checking by adding it to hack/.shellcheck_failures (if your reviewer is okay with it).' + echo + } >&2 + false +fi + +if [[ ${#not_failing[@]} -gt 0 ]]; then + { + echo "Some packages in hack/.shellcheck_failures are passing shellcheck. Please remove them." + echo + for f in "${not_failing[@]}"; do + echo " $f" + done + echo + } >&2 + false +fi + +# Check that all failing_packages actually still exist +gone=() +for f in "${failing_files[@]}"; do + kube::util::array_contains "$f" "${all_shell_scripts[@]}" || gone+=( "$f" ) +done + +if [[ ${#gone[@]} -gt 0 ]]; then + { + echo "Some files in hack/.shellcheck_failures do not exist anymore. Please remove them." + echo + for f in "${gone[@]}"; do + echo " $f" + done + echo + } >&2 + false +fi From e6db50df7eefbc7a0f116e5dff79b01d9cc22741 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Wed, 27 Mar 2019 17:39:43 +0100 Subject: [PATCH 018/107] check vendor directory This runs "dep check" to verify that the vendor directory is up-to-date and meets expectations (= done with dep >= 0.5.0). --- build.make | 10 ++++++++++ travis.yml | 5 +++++ 2 files changed, 15 insertions(+) diff --git a/build.make b/build.make index 8ca0b2c..b408406 100644 --- a/build.make +++ b/build.make @@ -117,6 +117,16 @@ test-fmt: false; \ fi +# This test only runs when dep >= 0.5 is installed, which is the case for the CI setup. +.PHONY: test-vendor +test: test-vendor +test-vendor: + @ echo; echo "### $@:" + @ case "$$(dep version 2>/dev/null | grep 'version *:')" in \ + *v0.[56789]*) dep check && echo "vendor up-to-date" || false;; \ + *) echo "skipping check, dep >= 0.5 required";; \ + esac + .PHONY: test-subtree test: test-subtree test-subtree: diff --git a/travis.yml b/travis.yml index b5a360a..1c05dfd 100644 --- a/travis.yml +++ b/travis.yml @@ -5,6 +5,11 @@ services: matrix: include: - go: 1.11.1 +before_script: +- mkdir -p bin +- wget https://github.com/golang/dep/releases/download/v0.5.1/dep-linux-amd64 -O bin/dep +- chmod u+x bin/dep +- export PATH=$PWD/bin:$PATH script: - make -k all test after_success: From 104a1ac9699f22e815c54afc2620aef85bf490e2 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Fri, 15 Mar 2019 11:08:53 +0100 Subject: [PATCH 019/107] build.make: avoid unit-testing E2E test suite In repos that have a test/e2e, that test suite should be run separately because it depends on a running cluster. --- build.make | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.make b/build.make index 8ca0b2c..6ea3279 100644 --- a/build.make +++ b/build.make @@ -98,7 +98,7 @@ test: test: test-go test-go: @ echo; echo "### $@:" - go test `go list ./... | grep -v 'vendor' $(TEST_GO_FILTER_CMD)` $(TESTARGS) + go test `go list ./... | grep -v -e 'vendor' -e '/test/e2e$$' $(TEST_GO_FILTER_CMD)` $(TESTARGS) .PHONY: test-vet test: test-vet From b2d25d4f4d7886b1d9ad40943dd06748e0f06874 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Fri, 15 Mar 2019 16:42:10 +0100 Subject: [PATCH 020/107] verify-shellcheck.sh: make it usable in csi-release-tools These are the modifications that were necessary to call this outside of Kubernetes. The support for excluding files from checking gets removed to simplify the script. It shouldn't be needed, because linting can be enabled after fixing whatever scripts might fail the check. --- util.sh | 691 ------------------------------------------- verify-shellcheck.sh | 71 +---- 2 files changed, 15 insertions(+), 747 deletions(-) diff --git a/util.sh b/util.sh index 31ce9fc..abeb1b2 100755 --- a/util.sh +++ b/util.sh @@ -39,31 +39,6 @@ kube::util::array_contains() { return 1 } -kube::util::wait_for_url() { - local url=$1 - local prefix=${2:-} - local wait=${3:-1} - local times=${4:-30} - local maxtime=${5:-1} - - command -v curl >/dev/null || { - kube::log::usage "curl must be installed" - exit 1 - } - - local i - for i in $(seq 1 "${times}"); do - local out - if out=$(curl --max-time "${maxtime}" -gkfs "${url}" 2>/dev/null); then - kube::log::status "On try ${i}, ${prefix}: ${out}" - return 0 - fi - sleep "${wait}" - done - kube::log::error "Timed out waiting for ${prefix} to answer at ${url}; tried ${times} waiting ${wait} between each" - return 1 -} - # Example: kube::util::trap_add 'echo "in trap DEBUG"' DEBUG # See: http://stackoverflow.com/questions/3338030/multiple-bash-traps-for-the-same-signal kube::util::trap_add() { @@ -93,422 +68,6 @@ kube::util::trap_add() { done } -# Opposite of kube::util::ensure-temp-dir() -kube::util::cleanup-temp-dir() { - rm -rf "${KUBE_TEMP}" -} - -# Create a temp dir that'll be deleted at the end of this bash session. -# -# Vars set: -# KUBE_TEMP -kube::util::ensure-temp-dir() { - if [[ -z ${KUBE_TEMP-} ]]; then - KUBE_TEMP=$(mktemp -d 2>/dev/null || mktemp -d -t kubernetes.XXXXXX) - kube::util::trap_add kube::util::cleanup-temp-dir EXIT - fi -} - -kube::util::host_os() { - local host_os - case "$(uname -s)" in - Darwin) - host_os=darwin - ;; - Linux) - host_os=linux - ;; - *) - kube::log::error "Unsupported host OS. Must be Linux or Mac OS X." - exit 1 - ;; - esac - echo "${host_os}" -} - -kube::util::host_arch() { - local host_arch - case "$(uname -m)" in - x86_64*) - host_arch=amd64 - ;; - i?86_64*) - host_arch=amd64 - ;; - amd64*) - host_arch=amd64 - ;; - aarch64*) - host_arch=arm64 - ;; - arm64*) - host_arch=arm64 - ;; - arm*) - host_arch=arm - ;; - i?86*) - host_arch=x86 - ;; - s390x*) - host_arch=s390x - ;; - ppc64le*) - host_arch=ppc64le - ;; - *) - kube::log::error "Unsupported host arch. Must be x86_64, 386, arm, arm64, s390x or ppc64le." - exit 1 - ;; - esac - echo "${host_arch}" -} - -# This figures out the host platform without relying on golang. We need this as -# we don't want a golang install to be a prerequisite to building yet we need -# this info to figure out where the final binaries are placed. -kube::util::host_platform() { - echo "$(kube::util::host_os)/$(kube::util::host_arch)" -} - -kube::util::find-binary-for-platform() { - local -r lookfor="$1" - local -r platform="$2" - local locations=( - "${KUBE_ROOT}/_output/bin/${lookfor}" - "${KUBE_ROOT}/_output/dockerized/bin/${platform}/${lookfor}" - "${KUBE_ROOT}/_output/local/bin/${platform}/${lookfor}" - "${KUBE_ROOT}/platforms/${platform}/${lookfor}" - ) - # Also search for binary in bazel build tree. - # The bazel go rules place some binaries in subtrees like - # "bazel-bin/source/path/linux_amd64_pure_stripped/binaryname", so make sure - # the platform name is matched in the path. - while IFS=$'\n' read -r location; do - locations+=("$location"); - done < <(find "${KUBE_ROOT}/bazel-bin/" -type f -executable \ - \( -path "*/${platform/\//_}*/${lookfor}" -o -path "*/${lookfor}" \) 2>/dev/null || true) - - # List most recently-updated location. - local -r bin=$( (ls -t "${locations[@]}" 2>/dev/null || true) | head -1 ) - echo -n "${bin}" -} - -kube::util::find-binary() { - kube::util::find-binary-for-platform "$1" "$(kube::util::host_platform)" -} - -# Run all known doc generators (today gendocs and genman for kubectl) -# $1 is the directory to put those generated documents -kube::util::gen-docs() { - local dest="$1" - - # Find binary - gendocs=$(kube::util::find-binary "gendocs") - genkubedocs=$(kube::util::find-binary "genkubedocs") - genman=$(kube::util::find-binary "genman") - genyaml=$(kube::util::find-binary "genyaml") - genfeddocs=$(kube::util::find-binary "genfeddocs") - - # TODO: If ${genfeddocs} is not used from anywhere (it isn't used at - # least from k/k tree), remove it completely. - kube::util::sourced_variable "${genfeddocs}" - - mkdir -p "${dest}/docs/user-guide/kubectl/" - "${gendocs}" "${dest}/docs/user-guide/kubectl/" - mkdir -p "${dest}/docs/admin/" - "${genkubedocs}" "${dest}/docs/admin/" "kube-apiserver" - "${genkubedocs}" "${dest}/docs/admin/" "kube-controller-manager" - "${genkubedocs}" "${dest}/docs/admin/" "cloud-controller-manager" - "${genkubedocs}" "${dest}/docs/admin/" "kube-proxy" - "${genkubedocs}" "${dest}/docs/admin/" "kube-scheduler" - "${genkubedocs}" "${dest}/docs/admin/" "kubelet" - "${genkubedocs}" "${dest}/docs/admin/" "kubeadm" - - mkdir -p "${dest}/docs/man/man1/" - "${genman}" "${dest}/docs/man/man1/" "kube-apiserver" - "${genman}" "${dest}/docs/man/man1/" "kube-controller-manager" - "${genman}" "${dest}/docs/man/man1/" "cloud-controller-manager" - "${genman}" "${dest}/docs/man/man1/" "kube-proxy" - "${genman}" "${dest}/docs/man/man1/" "kube-scheduler" - "${genman}" "${dest}/docs/man/man1/" "kubelet" - "${genman}" "${dest}/docs/man/man1/" "kubectl" - "${genman}" "${dest}/docs/man/man1/" "kubeadm" - - mkdir -p "${dest}/docs/yaml/kubectl/" - "${genyaml}" "${dest}/docs/yaml/kubectl/" - - # create the list of generated files - pushd "${dest}" > /dev/null || return 1 - touch docs/.generated_docs - find . -type f | cut -sd / -f 2- | LC_ALL=C sort > docs/.generated_docs - popd > /dev/null || return 1 -} - -# Removes previously generated docs-- we don't want to check them in. $KUBE_ROOT -# must be set. -kube::util::remove-gen-docs() { - if [ -e "${KUBE_ROOT}/docs/.generated_docs" ]; then - # remove all of the old docs; we don't want to check them in. - while read -r file; do - rm "${KUBE_ROOT}/${file}" 2>/dev/null || true - done <"${KUBE_ROOT}/docs/.generated_docs" - # The docs/.generated_docs file lists itself, so we don't need to explicitly - # delete it. - fi -} - -# Takes a group/version and returns the path to its location on disk, sans -# "pkg". E.g.: -# * default behavior: extensions/v1beta1 -> apis/extensions/v1beta1 -# * default behavior for only a group: experimental -> apis/experimental -# * Special handling for empty group: v1 -> api/v1, unversioned -> api/unversioned -# * Special handling for groups suffixed with ".k8s.io": foo.k8s.io/v1 -> apis/foo/v1 -# * Very special handling for when both group and version are "": / -> api -kube::util::group-version-to-pkg-path() { - local group_version="$1" - - while IFS=$'\n' read -r api; do - if [[ "${api}" = "${group_version/.*k8s.io/}" ]]; then - echo "vendor/k8s.io/api/${group_version/.*k8s.io/}" - return - fi - done < <(cd "${KUBE_ROOT}/staging/src/k8s.io/api" && find . -name types.go -exec dirname {} \; | sed "s|\./||g" | sort) - - # "v1" is the API GroupVersion - if [[ "${group_version}" == "v1" ]]; then - echo "vendor/k8s.io/api/core/v1" - return - fi - - # Special cases first. - # TODO(lavalamp): Simplify this by moving pkg/api/v1 and splitting pkg/api, - # moving the results to pkg/apis/api. - case "${group_version}" in - # both group and version are "", this occurs when we generate deep copies for internal objects of the legacy v1 API. - __internal) - echo "pkg/apis/core" - ;; - meta/v1) - echo "vendor/k8s.io/apimachinery/pkg/apis/meta/v1" - ;; - meta/v1beta1) - echo "vendor/k8s.io/apimachinery/pkg/apis/meta/v1beta1" - ;; - *.k8s.io) - echo "pkg/apis/${group_version%.*k8s.io}" - ;; - *.k8s.io/*) - echo "pkg/apis/${group_version/.*k8s.io/}" - ;; - *) - echo "pkg/apis/${group_version%__internal}" - ;; - esac -} - -# Takes a group/version and returns the swagger-spec file name. -# default behavior: extensions/v1beta1 -> extensions_v1beta1 -# special case for v1: v1 -> v1 -kube::util::gv-to-swagger-name() { - local group_version="$1" - case "${group_version}" in - v1) - echo "v1" - ;; - *) - echo "${group_version%/*}_${group_version#*/}" - ;; - esac -} - -# Returns the name of the upstream remote repository name for the local git -# repo, e.g. "upstream" or "origin". -kube::util::git_upstream_remote_name() { - git remote -v | grep fetch |\ - grep -E 'github.com[/:]kubernetes/kubernetes|k8s.io/kubernetes' |\ - head -n 1 | awk '{print $1}' -} - -# Ensures the current directory is a git tree for doing things like restoring or -# validating godeps -kube::util::create-fake-git-tree() { - local -r target_dir=${1:-$(pwd)} - - pushd "${target_dir}" >/dev/null || return 1 - git init >/dev/null - git config --local user.email "nobody@k8s.io" - git config --local user.name "$0" - git add . >/dev/null - git commit -q -m "Snapshot" >/dev/null - if (( ${KUBE_VERBOSE:-5} >= 6 )); then - kube::log::status "${target_dir} is now a git tree." - fi - popd >/dev/null || return 1 -} - -# Checks whether godep restore was run in the current GOPATH, i.e. that all referenced repos exist -# and are checked out to the referenced rev. -kube::util::godep_restored() { - local -r godeps_json=${1:-Godeps/Godeps.json} - local -r gopath=${2:-${GOPATH%:*}} - - kube::util::require-jq - - local root - local old_rev="" - while read -r path rev; do - rev="${rev//[\'\"]}" # remove quotes which are around revs sometimes - - if [[ "${rev}" == "${old_rev}" ]] && [[ "${path}" == "${root}"* ]]; then - # avoid checking the same git/hg root again - continue - fi - - root="${path}" - while [ "${root}" != "." ] && [ ! -d "${gopath}/src/${root}/.git" ] && [ ! -d "${gopath}/src/${root}/.hg" ]; do - root=$(dirname "${root}") - done - if [ "${root}" == "." ]; then - echo "No checkout of ${path} found in GOPATH \"${gopath}\"." 1>&2 - return 1 - fi - local head - if [ -d "${gopath}/src/${root}/.git" ]; then - head="$(cd "${gopath}/src/${root}" && git rev-parse HEAD)" - else - head="$(cd "${gopath}/src/${root}" && hg parent --template '{node}')" - fi - if [ "${head}" != "${rev}" ]; then - echo "Unexpected HEAD '${head}' at ${gopath}/src/${root}, expected '${rev}'." 1>&2 - return 1 - fi - old_rev="${rev}" - done < <(jq '.Deps|.[]|.ImportPath + " " + .Rev' -r < "${godeps_json}") - return 0 -} - -# Exits script if working directory is dirty. If it's run interactively in the terminal -# the user can commit changes in a second terminal. This script will wait. -kube::util::ensure_clean_working_dir() { - while ! git diff HEAD --exit-code &>/dev/null; do - echo -e "\nUnexpected dirty working directory:\n" - if tty -s; then - git status -s - else - git diff -a # be more verbose in log files without tty - exit 1 - fi | sed 's/^/ /' - echo -e "\nCommit your changes in another terminal and then continue here by pressing enter." - read -r - done 1>&2 -} - -# Ensure that the given godep version is installed and in the path. Almost -# nobody should use any version but the default. -# -# Sets: -# KUBE_GODEP: The path to the godep binary -# -kube::util::ensure_godep_version() { - local godep_target_version=${1:-"v80-k8s-r1"} # this version is known to work - - # If KUBE_GODEP is already set, and it's the right version, then use it. - if [[ -n "${KUBE_GODEP:-}" && "$(${KUBE_GODEP:?} version 2>/dev/null)" == *"godep ${godep_target_version}"* ]]; then - kube::log::status "Using ${KUBE_GODEP}" - return - fi - - # Otherwise, install forked godep - kube::log::status "Installing godep version ${godep_target_version}" - GOBIN="${KUBE_OUTPUT_BINPATH}" go install k8s.io/kubernetes/third_party/forked/godep - export KUBE_GODEP="${KUBE_OUTPUT_BINPATH}/godep" - kube::log::status "Installed ${KUBE_GODEP}" - - # Verify that the installed godep from fork is what we expect - if [[ "$(${KUBE_GODEP:?} version 2>/dev/null)" != *"godep ${godep_target_version}"* ]]; then - kube::log::error "Expected godep ${godep_target_version} from ${KUBE_GODEP}, got $(${KUBE_GODEP:?} version)" - return 1 - fi -} - -# Ensure that none of the staging repos is checked out in the GOPATH because this -# easily confused godep. -kube::util::ensure_no_staging_repos_in_gopath() { - kube::util::ensure_single_dir_gopath - local error=0 - for repo_file in "${KUBE_ROOT}"/staging/src/k8s.io/*; do - if [[ ! -d "${repo_file}" ]]; then - # not a directory or there were no files - continue; - fi - repo="$(basename "${repo_file}")" - if [ -e "${GOPATH}/src/k8s.io/${repo}" ]; then - echo "k8s.io/${repo} exists in GOPATH. Remove before running godep-save.sh." 1>&2 - error=1 - fi - done - if [ "${error}" = "1" ]; then - exit 1 - fi -} - -# Checks that the GOPATH is simple, i.e. consists only of one directory, not multiple. -kube::util::ensure_single_dir_gopath() { - if [[ "${GOPATH}" == *:* ]]; then - echo "GOPATH must consist of a single directory." 1>&2 - exit 1 - fi -} - -# Find the base commit using: -# $PULL_BASE_SHA if set (from Prow) -# current ref from the remote upstream branch -kube::util::base_ref() { - local -r git_branch=$1 - - if [[ -n ${PULL_BASE_SHA:-} ]]; then - echo "${PULL_BASE_SHA}" - return - fi - - full_branch="$(kube::util::git_upstream_remote_name)/${git_branch}" - - # make sure the branch is valid, otherwise the check will pass erroneously. - if ! git describe "${full_branch}" >/dev/null; then - # abort! - exit 1 - fi - - echo "${full_branch}" -} - -# Checks whether there are any files matching pattern $2 changed between the -# current branch and upstream branch named by $1. -# Returns 1 (false) if there are no changes -# 0 (true) if there are changes detected. -kube::util::has_changes() { - local -r git_branch=$1 - local -r pattern=$2 - local -r not_pattern=${3:-totallyimpossiblepattern} - - local base_ref - base_ref=$(kube::util::base_ref "${git_branch}") - echo "Checking for '${pattern}' changes against '${base_ref}'" - - # notice this uses ... to find the first shared ancestor - if git diff --name-only "${base_ref}...HEAD" | grep -v -E "${not_pattern}" | grep "${pattern}" > /dev/null; then - return 0 - fi - # also check for pending changes - if git status --porcelain | grep -v -E "${not_pattern}" | grep "${pattern}" > /dev/null; then - echo "Detected '${pattern}' uncommitted changes." - return 0 - fi - echo "No '${pattern}' changes detected." - return 1 -} - kube::util::download_file() { local -r url=$1 local -r destination_file=$2 @@ -528,152 +87,6 @@ kube::util::download_file() { return 1 } -# Test whether openssl is installed. -# Sets: -# OPENSSL_BIN: The path to the openssl binary to use -function kube::util::test_openssl_installed { - if ! openssl version >& /dev/null; then - echo "Failed to run openssl. Please ensure openssl is installed" - exit 1 - fi - - OPENSSL_BIN=$(command -v openssl) -} - -# creates a client CA, args are sudo, dest-dir, ca-id, purpose -# purpose is dropped in after "key encipherment", you usually want -# '"client auth"' -# '"server auth"' -# '"client auth","server auth"' -function kube::util::create_signing_certkey { - local sudo=$1 - local dest_dir=$2 - local id=$3 - local purpose=$4 - # Create client ca - ${sudo} /usr/bin/env bash -e < "${dest_dir}/${id}-ca-config.json" -EOF -} - -# signs a client certificate: args are sudo, dest-dir, CA, filename (roughly), username, groups... -function kube::util::create_client_certkey { - local sudo=$1 - local dest_dir=$2 - local ca=$3 - local id=$4 - local cn=${5:-$4} - local groups="" - local SEP="" - shift 5 - while [ -n "${1:-}" ]; do - groups+="${SEP}{\"O\":\"$1\"}" - SEP="," - shift 1 - done - ${sudo} /usr/bin/env bash -e < /dev/null -apiVersion: v1 -kind: Config -clusters: - - cluster: - certificate-authority: ${ca_file} - server: https://${api_host}:${api_port}/ - name: local-up-cluster -users: - - user: - token: ${token} - client-certificate: ${dest_dir}/client-${client_id}.crt - client-key: ${dest_dir}/client-${client_id}.key - name: local-up-cluster -contexts: - - context: - cluster: local-up-cluster - user: local-up-cluster - name: local-up-cluster -current-context: local-up-cluster -EOF - - # flatten the kubeconfig files to make them self contained - username=$(whoami) - ${sudo} /usr/bin/env bash -e < "/tmp/${client_id}.kubeconfig" - mv -f "/tmp/${client_id}.kubeconfig" "${dest_dir}/${client_id}.kubeconfig" - chown ${username} "${dest_dir}/${client_id}.kubeconfig" -EOF -} - -# Determines if docker can be run, failures may simply require that the user be added to the docker group. -function kube::util::ensure_docker_daemon_connectivity { - IFS=" " read -ra DOCKER <<< "${DOCKER_OPTS}" - # Expand ${DOCKER[@]} only if it's not unset. This is to work around - # Bash 3 issue with unbound variable. - DOCKER=(docker ${DOCKER[@]:+"${DOCKER[@]}"}) - if ! "${DOCKER[@]}" info > /dev/null 2>&1 ; then - cat <<'EOF' >&2 -Can't connect to 'docker' daemon. please fix and retry. - -Possible causes: - - Docker Daemon not started - - Linux: confirm via your init system - - macOS w/ docker-machine: run `docker-machine ls` and `docker-machine start ` - - macOS w/ Docker for Mac: Check the menu bar and start the Docker application - - DOCKER_HOST hasn't been set or is set incorrectly - - Linux: domain socket is used, DOCKER_* should be unset. In Bash run `unset ${!DOCKER_*}` - - macOS w/ docker-machine: run `eval "$(docker-machine env )"` - - macOS w/ Docker for Mac: domain socket is used, DOCKER_* should be unset. In Bash run `unset ${!DOCKER_*}` - - Other things to check: - - Linux: User isn't in 'docker' group. Add and relogin. - - Something like 'sudo usermod -a -G docker ${USER}' - - RHEL7 bug and workaround: https://bugzilla.redhat.com/show_bug.cgi?id=1119282#c8 -EOF - return 1 - fi -} - # Wait for background jobs to finish. Return with # an error status if any of the jobs failed. kube::util::wait-for-jobs() { @@ -696,101 +109,6 @@ function kube::util::join { echo "$*" } -# Downloads cfssl/cfssljson into $1 directory if they do not already exist in PATH -# -# Assumed vars: -# $1 (cfssl directory) (optional) -# -# Sets: -# CFSSL_BIN: The path of the installed cfssl binary -# CFSSLJSON_BIN: The path of the installed cfssljson binary -# -function kube::util::ensure-cfssl { - if command -v cfssl &>/dev/null && command -v cfssljson &>/dev/null; then - CFSSL_BIN=$(command -v cfssl) - CFSSLJSON_BIN=$(command -v cfssljson) - return 0 - fi - - host_arch=$(kube::util::host_arch) - - if [[ "${host_arch}" != "amd64" ]]; then - echo "Cannot download cfssl on non-amd64 hosts and cfssl does not appear to be installed." - echo "Please install cfssl and cfssljson and verify they are in \$PATH." - echo "Hint: export PATH=\$PATH:\$GOPATH/bin; go get -u github.com/cloudflare/cfssl/cmd/..." - exit 1 - fi - - # Create a temp dir for cfssl if no directory was given - local cfssldir=${1:-} - if [[ -z "${cfssldir}" ]]; then - kube::util::ensure-temp-dir - cfssldir="${KUBE_TEMP}/cfssl" - fi - - mkdir -p "${cfssldir}" - pushd "${cfssldir}" > /dev/null || return 1 - - echo "Unable to successfully run 'cfssl' from ${PATH}; downloading instead..." - kernel=$(uname -s) - case "${kernel}" in - Linux) - curl --retry 10 -L -o cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 - curl --retry 10 -L -o cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 - ;; - Darwin) - curl --retry 10 -L -o cfssl https://pkg.cfssl.org/R1.2/cfssl_darwin-amd64 - curl --retry 10 -L -o cfssljson https://pkg.cfssl.org/R1.2/cfssljson_darwin-amd64 - ;; - *) - echo "Unknown, unsupported platform: ${kernel}." >&2 - echo "Supported platforms: Linux, Darwin." >&2 - exit 2 - esac - - chmod +x cfssl || true - chmod +x cfssljson || true - - CFSSL_BIN="${cfssldir}/cfssl" - CFSSLJSON_BIN="${cfssldir}/cfssljson" - if [[ ! -x ${CFSSL_BIN} || ! -x ${CFSSLJSON_BIN} ]]; then - echo "Failed to download 'cfssl'. Please install cfssl and cfssljson and verify they are in \$PATH." - echo "Hint: export PATH=\$PATH:\$GOPATH/bin; go get -u github.com/cloudflare/cfssl/cmd/..." - exit 1 - fi - popd > /dev/null || return 1 -} - -# kube::util::ensure_dockerized -# Confirms that the script is being run inside a kube-build image -# -function kube::util::ensure_dockerized { - if [[ -f /kube-build-image ]]; then - return 0 - else - echo "ERROR: This script is designed to be run inside a kube-build container" - exit 1 - fi -} - -# kube::util::ensure-gnu-sed -# Determines which sed binary is gnu-sed on linux/darwin -# -# Sets: -# SED: The name of the gnu-sed binary -# -function kube::util::ensure-gnu-sed { - if LANG=C sed --help 2>&1 | grep -q GNU; then - SED="sed" - elif command -v gsed &>/dev/null; then - SED="gsed" - else - kube::log::error "Failed to find GNU sed as sed or gsed. If you are on Mac: brew install gnu-sed." >&2 - return 1 - fi - kube::util::sourced_variable "${SED}" -} - # kube::util::check-file-in-alphabetical-order # Check that the file is in alphabetical order # @@ -808,15 +126,6 @@ function kube::util::check-file-in-alphabetical-order { fi } -# kube::util::require-jq -# Checks whether jq is installed. -function kube::util::require-jq { - if ! command -v jq &>/dev/null; then - echo "jq not found. Please install." 1>&2 - return 1 - fi -} - # Some useful colors. if [[ -z "${color_start-}" ]]; then declare -r color_start="\033[" diff --git a/verify-shellcheck.sh b/verify-shellcheck.sh index 1b882fa..fd28021 100755 --- a/verify-shellcheck.sh +++ b/verify-shellcheck.sh @@ -18,9 +18,12 @@ set -o errexit set -o nounset set -o pipefail -KUBE_ROOT=$(dirname "${BASH_SOURCE[0]}")/.. -source "${KUBE_ROOT}/hack/lib/init.sh" -source "${KUBE_ROOT}/hack/lib/util.sh" +# The csi-release-tools directory. +TOOLS="$(dirname "${BASH_SOURCE[0]}")" +. "${TOOLS}/util.sh" + +# Directory to check. Default is the parent of the tools themselves. +ROOT="${1:-${TOOLS}/..}" # required version for this script, if not installed on the host we will # use the official docker image instead. keep this in sync with SHELLCHECK_IMAGE @@ -56,15 +59,15 @@ create_container () { # we're done. # This is incredibly much faster than creating a container for each shellcheck # call ... - docker run --name "${SHELLCHECK_CONTAINER}" -d --rm -v "${KUBE_ROOT}:${KUBE_ROOT}" -w "${KUBE_ROOT}" --entrypoint="sleep" "${SHELLCHECK_IMAGE}" 2147483647 + docker run --name "${SHELLCHECK_CONTAINER}" -d --rm -v "${ROOT}:${ROOT}" -w "${ROOT}" --entrypoint="sleep" "${SHELLCHECK_IMAGE}" 2147483647 } # removes the shellcheck container remove_container () { docker rm -f "${SHELLCHECK_CONTAINER}" &> /dev/null || true } -# ensure we're linting the k8s source tree -cd "${KUBE_ROOT}" +# ensure we're linting the source tree +cd "${ROOT}" # find all shell scripts excluding ./_*, ./.git/*, ./vendor*, # and anything git-ignored @@ -78,16 +81,6 @@ done < <(find . -name "*.sh" \ -path ./vendor\* \ \)) -# make sure known failures are sorted -failure_file="${KUBE_ROOT}/hack/.shellcheck_failures" -kube::util::check-file-in-alphabetical-order "${failure_file}" - -# load known failure files -failing_files=() -while IFS=$'\n' read -r script; - do failing_files+=("$script"); -done < <(cat "${failure_file}") - # detect if the host machine has the required shellcheck version installed # if so, we will use that instead. HAVE_SHELLCHECK=false @@ -119,7 +112,6 @@ fi # lint each script, tracking failures errors=() -not_failing=() for f in "${all_shell_scripts[@]}"; do set +o errexit if ${HAVE_SHELLCHECK}; then @@ -129,18 +121,14 @@ for f in "${all_shell_scripts[@]}"; do shellcheck --exclude="${SHELLCHECK_DISABLED}" "${f}") fi set -o errexit - kube::util::array_contains "${f}" "${failing_files[@]}" && in_failing=$? || in_failing=$? - if [[ -n "${failedLint}" ]] && [[ "${in_failing}" -ne "0" ]]; then - errors+=( "${failedLint}" ) - fi - if [[ -z "${failedLint}" ]] && [[ "${in_failing}" -eq "0" ]]; then - not_failing+=( "${f}" ) + if [[ -n "${failedLint}" ]]; then + errors+=( "${failedLint}" ) fi done # Check to be sure all the packages that should pass lint are. if [ ${#errors[@]} -eq 0 ]; then - echo 'Congratulations! All shell files are passing lint (excluding those in hack/.shellcheck_failures).' + echo 'Congratulations! All shell files are passing lint.' else { echo "Errors from shellcheck:" @@ -149,38 +137,9 @@ else done echo echo 'Please review the above warnings. You can test via "./hack/verify-shellcheck"' - echo 'If the above warnings do not make sense, you can exempt this package from shellcheck' - echo 'checking by adding it to hack/.shellcheck_failures (if your reviewer is okay with it).' - echo - } >&2 - false -fi - -if [[ ${#not_failing[@]} -gt 0 ]]; then - { - echo "Some packages in hack/.shellcheck_failures are passing shellcheck. Please remove them." - echo - for f in "${not_failing[@]}"; do - echo " $f" - done - echo - } >&2 - false -fi - -# Check that all failing_packages actually still exist -gone=() -for f in "${failing_files[@]}"; do - kube::util::array_contains "$f" "${all_shell_scripts[@]}" || gone+=( "$f" ) -done - -if [[ ${#gone[@]} -gt 0 ]]; then - { - echo "Some files in hack/.shellcheck_failures do not exist anymore. Please remove them." - echo - for f in "${gone[@]}"; do - echo " $f" - done + echo 'If the above warnings do not make sense, you can exempt them from shellcheck' + echo 'checking by adding the "shellcheck disable" directive' + echo '(https://github.com/koalaman/shellcheck/wiki/Directive#disable).' echo } >&2 false From 6c7ba1be0fbb012b6b3d6d204fcc363164bce12d Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Tue, 26 Mar 2019 13:09:00 +0100 Subject: [PATCH 021/107] build.make: integrate shellcheck into "make test" By default this only tests the scripts inside the "release-tools" directory, which is useful when making experimental changes to them in a component that uses csi-release-tools. But a component can also enable checking for other directories. --- README.md | 12 ++++++++++++ build.make | 15 +++++++++++++++ 2 files changed, 27 insertions(+) diff --git a/README.md b/README.md index 56d2248..cb02815 100644 --- a/README.md +++ b/README.md @@ -49,3 +49,15 @@ Cheat sheet: - `git subtree add --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master` - add release tools to a repo which does not have them yet (only once) - `git subtree pull --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master` - update local copy to latest upstream (whenever upstream changes) - edit, `git commit`, `git subtree push --prefix=release-tools git@github.com:/csi-release-tools.git ` - push to a new branch before submitting a PR + +verify-shellcheck.sh +-------------------- + +The [verify-shellcheck.sh](./verify-shellcheck.sh) script in this repo +is a stripped down copy of the [corresponding +script](https://github.com/kubernetes/kubernetes/blob/release-1.14/hack/verify-shellcheck.sh) +in the Kubernetes repository. It can be used to check for certain +errors shell scripts, like missing quotation marks. The default +`test-shellcheck` target in [build.make](./build.make) only checks the +scripts in this directory. Components can add more directories to +`TEST_SHELLCHECK_DIRS` to check also other scripts. diff --git a/build.make b/build.make index 6ea3279..3cac9d3 100644 --- a/build.make +++ b/build.make @@ -122,3 +122,18 @@ test: test-subtree test-subtree: @ echo; echo "### $@:" ./release-tools/verify-subtree.sh release-tools + +# Components can extend the set of directories which must pass shellcheck. +# The default is to check only the release-tools directory itself. +TEST_SHELLCHECK_DIRS=release-tools +.PHONY: test-shellcheck +test: test-shellcheck +test-shellcheck: + @ echo; echo "### $@:" + @ ret=0; \ + for dir in $(abspath $(TEST_SHELLCHECK_DIRS)); do \ + echo; \ + echo "$$dir:"; \ + ./release-tools/verify-shellcheck.sh "$$dir" || ret=1; \ + done; \ + return $$ret From 55212ff2b4f6a83c22cf2344b101d31cd11ec271 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Fri, 15 Mar 2019 16:45:16 +0100 Subject: [PATCH 022/107] initial Prow test job This enables testing of other repos and of this repo itself inside Prow. Currently supported is unit testing ("make test") and E2E testing (either via a local test suite or the Kubernetes E2E test suite applied to the hostpath driver example deployment). The script passes shellcheck and uses Prow to verify that for future PRs. --- .prow.sh | 7 + README.md | 45 +++ filter-junit.go | 133 +++++++ prow.sh | 916 ++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 1101 insertions(+) create mode 100755 .prow.sh create mode 100644 filter-junit.go create mode 100755 prow.sh diff --git a/.prow.sh b/.prow.sh new file mode 100755 index 0000000..b18c535 --- /dev/null +++ b/.prow.sh @@ -0,0 +1,7 @@ +#! /bin/bash -e +# +# This is for testing csi-release-tools itself in Prow. All other +# repos use prow.sh for that, but as csi-release-tools isn't a normal +# repo with some Go code in it, it has a custom Prow test script. + +./verify-shellcheck.sh "$(pwd)" diff --git a/README.md b/README.md index cb02815..bc061ae 100644 --- a/README.md +++ b/README.md @@ -61,3 +61,48 @@ errors shell scripts, like missing quotation marks. The default `test-shellcheck` target in [build.make](./build.make) only checks the scripts in this directory. Components can add more directories to `TEST_SHELLCHECK_DIRS` to check also other scripts. + +End-to-end testing +------------------ + +A repo that wants to opt into testing via Prow must set up a top-level +`.prow.sh`. Typically that will source `prow.sh` and then transfer +control to it: + +``` bash +#! /bin/bash -e + +. release-tools/prow.sh +main +``` + +All Kubernetes-CSI repos are expected to switch to Prow. For details +on what is enabled in Prow, see +https://github.com/kubernetes/test-infra/tree/master/config/jobs/kubernetes-csi + +Test results for periodic jobs are visible in +https://testgrid.k8s.io/sig-storage-csi + +It is possible to reproduce the Prow testing locally on a suitable machine: +- Linux host +- Docker installed +- code to be tested checkout out in `$GOPATH/src/` +- `cd $GOPATH/src/ && ./.prow.sh` + +Beware that the script intentionally doesn't clean up after itself and +modifies the content of `$GOPATH`, in particular the `kubernetes` and +`kind` repositories there. Better run it in an empty, disposable +`$GOPATH`. + +When it terminates, the following command can be used to get access to +the Kubernetes cluster that was brought up for testing (assuming that +this step succeeded): + + export KUBECONFIG="$(kind get kubeconfig-path --name="csi-prow")" + +It is possible to control the execution via environment variables. See +`prow.sh` for details. Particularly useful is testing against different +Kubernetes releases: + + CSI_PROW_KUBERNETES_VERSION=1.13.3 ./.prow.sh + CSI_PROW_KUBERNETES_VERSION=latest ./.prow.sh diff --git a/filter-junit.go b/filter-junit.go new file mode 100644 index 0000000..2f51be0 --- /dev/null +++ b/filter-junit.go @@ -0,0 +1,133 @@ +/* +Copyright 2019 The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +/* + * This command filters a JUnit file such that only tests with a name + * matching a regular expression are passed through. By concatenating + * multiple input files it is possible to merge them into a single file. + */ +package main + +import ( + "encoding/xml" + "flag" + "io/ioutil" + "os" + "regexp" +) + +var ( + output = flag.String("o", "-", "junit file to write, - for stdout") + tests = flag.String("t", "", "regular expression matching the test names that are to be included in the output") +) + +/* + * TestSuite represents a JUnit file. Due to how encoding/xml works, we have + * represent all fields that we want to be passed through. It's therefore + * not a complete solution, but good enough for Ginkgo + Spyglass. + */ +type TestSuite struct { + XMLName string `xml:"testsuite"` + TestCases []TestCase `xml:"testcase"` +} + +type TestCase struct { + Name string `xml:"name,attr"` + Time string `xml:"time,attr"` + SystemOut string `xml:"system-out,omitempty"` + Failure string `xml:"failure,omitempty"` + Skipped SkipReason `xml:"skipped,omitempty"` +} + +// SkipReason deals with the special : +// if present, we must re-encode it, even if empty. +type SkipReason string + +func (s *SkipReason) UnmarshalText(text []byte) error { + *s = SkipReason(text) + if *s == "" { + *s = " " + } + return nil +} + +func (s SkipReason) MarshalText() ([]byte, error) { + if s == " " { + return []byte{}, nil + } + return []byte(s), nil +} + +func main() { + var junit TestSuite + var data []byte + + flag.Parse() + + re := regexp.MustCompile(*tests) + + // Read all input files. + for _, input := range flag.Args() { + if input == "-" { + if _, err := os.Stdin.Read(data); err != nil { + panic(err) + } + } else { + var err error + data, err = ioutil.ReadFile(input) + if err != nil { + panic(err) + } + } + if err := xml.Unmarshal(data, &junit); err != nil { + panic(err) + } + } + + // Keep only matching testcases. Testcases skipped in all test runs are only stored once. + filtered := map[string]TestCase{} + for _, testcase := range junit.TestCases { + if !re.MatchString(testcase.Name) { + continue + } + entry, ok := filtered[testcase.Name] + if !ok || // not present yet + entry.Skipped != "" && testcase.Skipped == "" { // replaced skipped test with real test run + filtered[testcase.Name] = testcase + } + } + junit.TestCases = nil + for _, testcase := range filtered { + junit.TestCases = append(junit.TestCases, testcase) + } + + // Re-encode. + data, err := xml.MarshalIndent(junit, "", " ") + if err != nil { + panic(err) + } + + // Write to output. + if *output == "-" { + if _, err := os.Stdout.Write(data); err != nil { + panic(err) + } + } else { + if err := ioutil.WriteFile(*output, data, 0644); err != nil { + panic(err) + } + } +} diff --git a/prow.sh b/prow.sh new file mode 100755 index 0000000..17e8ae4 --- /dev/null +++ b/prow.sh @@ -0,0 +1,916 @@ +#! /bin/bash +# +# Copyright 2019 The Kubernetes Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# This script runs inside a Prow job. It runs unit tests ("make test") +# and E2E testing. This E2E testing covers different scenarios (see +# https://github.com/kubernetes/enhancements/pull/807): +# - running the stable hostpath example against a Kubernetes release +# - running the canary hostpath example against a Kubernetes release +# - building the component in the current repo and running the +# stable hostpath example with that one component replaced against +# a Kubernetes release +# +# The intended usage of this script is that individual repos import +# csi-release-tools, then link their top-level prow.sh to this or +# include it in that file. When including it, several of the variables +# can be overridden in the top-level prow.sh to customize the script +# for the repo. +# +# The expected environment is: +# - $GOPATH/src/ for the repository that is to be tested, +# with PR branch merged (when testing a PR) +# - running on linux-amd64 +# - bazel installed (when testing against Kubernetes master), must be recent +# enough for Kubernetes master +# - kind (https://github.com/kubernetes-sigs/kind) installed +# - optional: Go already installed + +# Sets the default value for a variable if not set already and logs the value. +# Any variable set this way is usually something that a repo's .prow.sh +# or the job can set. +configvar () { + # Ignore: Word is of the form "A"B"C" (B indicated). Did you mean "ABC" or "A\"B\"C"? + # shellcheck disable=SC2140 + eval : \$\{"$1":="\$2"\} + eval echo "\$3:" "$1=\${$1}" +} + +# Go versions can be specified seperately for different tasks +# If the pre-installed Go is missing or a different +# version, the required version here will get installed +# from https://golang.org/dl/. +configvar CSI_PROW_GO_VERSION_BUILD 1.11.4 "Go version for building the component" # depends on component's source code +configvar CSI_PROW_GO_VERSION_K8S 1.12.1 "Go version for building Kubernetes for the test cluster" # depends on Kubernetes version +configvar CSI_PROW_GO_VERSION_E2E 1.12.1 "Go version for building the Kubernetes E2E test suite" # depends on CSI_PROW_E2E settings below +configvar CSI_PROW_GO_VERSION_SANITY "${CSI_PROW_GO_VERSION_BUILD}" "Go version for building the csi-sanity test suite" # depends on CSI_PROW_SANITY settings below +configvar CSI_PROW_GO_VERSION_KIND "${CSI_PROW_GO_VERSION_BUILD}" "Go version for building 'kind'" # depends on CSI_PROW_KIND_VERSION below +configvar CSI_PROW_GO_VERSION_GINKGO "${CSI_PROW_GO_VERSION_BUILD}" "Go version for building ginkgo" # depends on CSI_PROW_GINKGO_VERSION below + +# kind version to use. If the pre-installed version is different, +# the desired version is downloaded from https://github.com/kubernetes-sigs/kind/releases/download/ +# (if available), otherwise it is built from source. +configvar CSI_PROW_KIND_VERSION 0.2.1 "kind" + +# ginkgo test runner version to use. If the pre-installed version is +# different, the desired version is built from source. +configvar CSI_PROW_GINKGO_VERSION v1.7.0 "Ginkgo" + +# Ginkgo runs the E2E test in parallel. The default is based on the number +# of CPUs, but typically this can be set to something higher in the job. +configvar CSI_PROW_GINKO_PARALLEL "-p" "Ginko parallelism parameter(s)" + +# Enables building the code in the repository. On by default, can be +# disabled in jobs which only use pre-built components. +configvar CSI_PROW_BUILD_JOB true "building code in repo enabled" + +# Kubernetes version to test against. This must be a version number +# (like 1.13.3) for which there is a pre-built kind image (see +# https://hub.docker.com/r/kindest/node/tags), "latest" (builds +# Kubernetes from the master branch) or "release-x.yy" (builds +# Kubernetes from a release branch). +# +# This can also be a version that was not released yet at the time +# that the settings below were chose. The script will then +# use the same settings as for "latest" Kubernetes. This works +# as long as there are no breaking changes in Kubernetes, like +# deprecating or changing the implementation of an alpha feature. +configvar CSI_PROW_KUBERNETES_VERSION 1.13.3 "Kubernetes" + +# CSI_PROW_KUBERNETES_VERSION reduced to first two version numbers and +# with underscore (1_13 instead of 1.13.3) and in uppercase (LATEST +# instead of latest). +# +# This is used to derive the right defaults for the variables below +# when a Prow job just defines the Kubernetes version. +csi_prow_kubernetes_version_suffix="$(echo "${CSI_PROW_KUBERNETES_VERSION}" | tr . _ | tr '[:lower:]' '[:upper:]' | sed -e 's/^RELEASE-//' -e 's/\([0-9]*\)_\([0-9]*\).*/\1_\2/')" + +# Work directory. It has to allow running executables, therefore /tmp +# is avoided. Cleaning up after the script is intentionally left to +# the caller. +configvar CSI_PROW_WORK "$(mkdir -p "$GOPATH/pkg" && mktemp -d "$GOPATH/pkg/csiprow.XXXXXXXXXX")" "work directory" + +# The hostpath deployment script is searched for in several places. +# +# - The "deploy" directory in the current repository: this is useful +# for the situation that a component becomes incompatible with the +# shared deployment, because then it can (temporarily!) provide its +# own example until the shared one can be updated; it's also how +# csi-driver-host-path itself provides the example. +# +# - CSI_PROW_HOSTPATH_VERSION of the CSI_PROW_HOSTPATH_REPO is checked +# out: this allows other repos to reference a version of the example +# that is known to be compatible. +# +# - The csi-driver-host-path/deploy directory has multiple sub-directories, +# each with different deployments (stable set of images for Kubernetes 1.13, +# stable set of images for Kubernetes 1.14, canary for latest Kubernetes, etc.). +# This is necessary because there may be incompatible changes in the +# "API" of a component (for example, its command line options or RBAC rules) +# or in its support for different Kubernetes versions (CSIDriverInfo as +# CRD in Kubernetes 1.13 vs builtin API in Kubernetes 1.14). +# +# When testing an update for a component in a PR job, the +# CSI_PROW_DEPLOYMENT variable can be set in the +# .prow.sh of each component when there are breaking changes +# that require using a non-default deployment. The default +# is a deployment named "kubernetes-x.yy" (if available), +# otherwise "kubernetes-latest". +# "none" disables the deployment of the hostpath driver. +# +# When no deploy script is found (nothing in `deploy` directory, +# CSI_PROW_HOSTPATH_REPO=none), nothing gets deployed. +# +# TODO: merge https://github.com/kubernetes-csi/csi-driver-host-path/pull/29, switch to revision from master +configvar CSI_PROW_HOSTPATH_VERSION deployment "hostpath driver" +configvar CSI_PROW_HOSTPATH_REPO https://github.com/pohly/csi-driver-host-path "hostpath repo" +# Ignore 'See if you can use ${variable//search/replace} instead.' +# shellcheck disable=SC2001 +configvar CSI_PROW_DEPLOYMENT "" "deployment" + +# If CSI_PROW_HOSTPATH_CANARY is set (typically to "canary", but also +# "1.0-canary"), then all image versions are replaced with that +# version tag. +configvar CSI_PROW_HOSTPATH_CANARY "" "hostpath image" + +# The E2E testing can come from an arbitrary repo. The expectation is that +# the repo supports "go test ./test/e2e -args --storage.testdriver" (https://github.com/kubernetes/kubernetes/pull/72836) +# after setting KUBECONFIG. As a special case, if the repository is Kubernetes, +# then `make WHAT=test/e2e/e2e.test` is called first to ensure that +# all generated files are present. +# +# CSI_PROW_E2E_REPO=none disables E2E testing. +configvar CSI_PROW_E2E_VERSION v1.14.0 "E2E version" +configvar CSI_PROW_E2E_REPO https://github.com/kubernetes/kubernetes "E2E repo" +configvar CSI_PROW_E2E_IMPORT_PATH k8s.io/kubernetes "E2E package" + +# csi-sanity testing from the csi-test repo can be run against the installed +# CSI driver. For this to work, deploying the driver must expose the Unix domain +# csi.sock as a TCP service for use by the csi-sanity command, which runs outside +# of the cluster. The alternative would have been to (cross-)compile csi-sanity +# and install it inside the cluster, which is not necessarily easier. +configvar CSI_PROW_SANITY_REPO https://github.com/kubernetes-csi/csi-test "csi-test repo" +configvar CSI_PROW_SANITY_VERSION 5421d9f3c37be3b95b241b44a094a3db11bee789 "csi-test version" # latest master +configvar CSI_PROW_SANITY_IMPORT_PATH github.com/kubernetes-csi/csi-test "csi-test package" +configvar CSI_PROW_SANITY_SERVICE "hostpath-service" "Kubernetes TCP service name that exposes csi.sock" +configvar CSI_PROW_SANITY_POD "csi-hostpathplugin-0" "Kubernetes pod with CSI driver" +configvar CSI_PROW_SANITY_CONTAINER "hostpath" "Kubernetes container with CSI driver" + +# Each job can run one or more of the following tests, identified by +# a single word: +# - unit testing +# - parallel excluding alpha features +# - serial excluding alpha features +# - parallel, only alpha feature +# - serial, only alpha features +# - sanity +# +# Sanity testing with csi-sanity only covers the CSI driver itself and thus +# is off by default. A CSI driver can change that default in its .prow.sh +# by setting CSI_PROW_TESTS_SANITY. +configvar CSI_PROW_TESTS "unit parallel serial parallel-alpha serial-alpha ${CSI_PROW_TESTS_SANITY}" "tests to run" +test_enabled () { + echo "${CSI_PROW_TESTS}" | grep -q -w -e "$1" +} + +# Serial vs. parallel is always determined by these regular expressions. +# Individual regular expressions are seperated by spaces for readability +# and expected to not contain spaces. Use dots instead. The complete +# regex for Ginkgo will be created by joining the individual terms. +configvar CSI_PROW_E2E_SERIAL '\[Serial\] \[Disruptive\]' "tags for serial E2E tests" +regex_join () { + echo "$@" | sed -e 's/ */|/g' -e 's/^|*//' -e 's/|*$//' -e 's/^$/this-matches-nothing/g' +} + +# Prints the value of a variable + version suffix, falling back to variable + "LATEST". +get_versioned_variable () { + local var="$1" + local version="$2" + local value + + eval value="\${${var}_${version}}" + if ! [ "$value" ]; then + eval value="\${${var}_LATEST}" + fi + echo "$value" +} + +# Which tests are alpha depends on the Kubernetes version. The same +# E2E test suite is used for all Kubernetes versions, including older +# Kubernetes. +# +# Feature tags in the test are set based on what is an alpha +# feature in the Kubernetes version that contains the E2E test suite's +# source code. +# +# So when testing against an older Kubernetes release, some tests +# might be enabled by default which don't pass for that older +# Kubernetes version. In that case, the +# CSI_PROW_E2E_ALPHA_ variable cannot just +# be based on the `Feature` tag, it also must contain the names +# of tests that used to be alpha but no longer have that tag. +# +# is just major+minor version separated by +# underscore. "latest" matches master. It is also used for +# unknown versions, so when master gets released there is +# no need to add a new versioned variable. That only needs to be +# done when updating something in the configuration that leads +# to "latest" no longer being suitable. +configvar CSI_PROW_E2E_ALPHA_1_13 '\[Feature: \[Testpattern:.Dynamic.PV..block.volmode.\] should.create.and.delete.block.persistent.volumes' "alpha tests for Kubernetes 1.13" # Raw block was an alpha feature in 1.13. +configvar CSI_PROW_E2E_ALPHA_LATEST '\[Feature:' "alpha tests for Kubernetes master" +configvar CSI_PROW_E2E_ALPHA "$(get_versioned_variable CSI_PROW_E2E_ALPHA "${csi_prow_kubernetes_version_suffix}")" "alpha tests" + +# After the parallel E2E test without alpha features, a test cluster +# with alpha features is brought up and tests that were previously +# disabled are run. The alpha gates in each release have to be listed +# explicitly. If none are set (= variable empty), alpha testing +# is skipped. +# +# Testing against "latest" Kubernetes is problematic because some alpha +# feature which used to work might stop working or change their behavior +# such that the current tests no longer pass. If that happens, +# kubernetes-csi components must be updated, either by disabling +# the failing test for "latest" or by updating the test and not running +# it anymore for older releases. +configvar CSI_PROW_E2E_ALPHA_GATES_1_13 'VolumeSnapshotDataSource=true,BlockVolume=true,CSIBlockVolume=true' "alpha feature gates for Kubernetes 1.13" +configvar CSI_PROW_E2E_ALPHA_GATES_LATEST 'VolumeSnapshotDataSource=true' "alpha feature gates for latest Kubernetes" +configvar CSI_PROW_E2E_ALPHA_GATES "$(get_versioned_variable CSI_PROW_E2E_ALPHA_GATES "${csi_prow_kubernetes_version_suffix}")" "alpha E2E feature gates" + +# Some tests are known to be unusable in a KinD cluster. For example, +# stopping kubelet with "ssh systemctl stop kubelet" simply +# doesn't work. Such tests should be written in a way that they verify +# whether they can run with the current cluster provider, but until +# they are, we filter them out by name. Like the other test selection +# variables, this is again a space separated list of regular expressions. +configvar CSI_PROW_E2E_SKIP 'while.kubelet.is.down.*Disruptive' "tests that need to be skipped" + +# This is the directory for additional result files. Usually set by Prow, but +# if not (for example, when invoking manually) it defaults to the work directory. +configvar ARTIFACTS "${CSI_PROW_WORK}/artifacts" "artifacts" +mkdir -p "${ARTIFACTS}" + +RELEASE_TOOLS_ROOT="$(realpath "$(dirname "${BASH_SOURCE[0]}")")" +REPO_DIR="$(pwd)" + +run () { + echo "$(date) $(go version | sed -e 's/.*version \(go[^ ]*\).*/\1/') $(if [ "$(pwd)" != "${REPO_DIR}" ]; then pwd; fi)\$" "$@" >&2 + "$@" +} + +info () { + echo >&2 INFO: "$@" +} + +warn () { + echo >&2 WARNING: "$@" +} + +die () { + echo >&2 ERROR: "$@" + exit 1 +} + +# For additional tools. +CSI_PROW_BIN="${CSI_PROW_WORK}/bin" +mkdir -p "${CSI_PROW_BIN}" +PATH="${CSI_PROW_BIN}:$PATH" + +# Ensure that PATH has the desired version of the Go tools, then run command given as argument. +run_with_go () { + local version + version="$1" + shift + + if go version 2>/dev/null | grep -q "go$version"; then + run "$@" + else + if ! [ -d "${CSI_PROW_WORK}/go-$version" ]; then + run curl --fail --location "https://dl.google.com/go/go$version.linux-amd64.tar.gz" | tar -C "${CSI_PROW_WORK}" -zxf - || die "installation of Go $version failed" + mv "${CSI_PROW_WORK}/go" "${CSI_PROW_WORK}/go-$version" + fi + PATH="${CSI_PROW_WORK}/go-$version/bin:$PATH" run "$@" + fi +} + +# Ensure that we have the desired version of kind. +install_kind () { + if kind --version 2>/dev/null | grep -q " ${CSI_PROW_KIND_VERSION}$"; then + return + fi + if run curl --fail --location -o "${CSI_PROW_WORK}/bin/kind" "https://github.com/kubernetes-sigs/kind/releases/download/${CSI_PROW_KIND_VERSION}/kind-linux-amd64"; then + chmod u+x "${CSI_PROW_WORK}/bin/kind" + else + git_checkout https://github.com/kubernetes-sigs/kind "$GOPATH/src/sigs.k8s.io/kind" "${CSI_PROW_KIND_VERSION}" --depth=1 && + run_with_go "${CSI_PROW_GO_VERSION_KIND}" go build -o "${CSI_PROW_WORK}/bin/kind" sigs.k8s.io/kind + fi +} + +# Ensure that we have the desired version of the ginkgo test runner. +install_ginkgo () { + # CSI_PROW_GINKGO_VERSION contains the tag with v prefix, the command line output does not. + if [ "v$(ginkgo version 2>/dev/null | sed -e 's/.* //')" = "${CSI_PROW_GINKGO_VERSION}" ]; then + return + fi + git_checkout https://github.com/onsi/ginkgo "$GOPATH/src/github.com/onsi/ginkgo" "${CSI_PROW_GINKGO_VERSION}" --depth=1 && + # We have to get dependencies and hence can't call just "go build". + run_with_go "${CSI_PROW_GO_VERSION_GINKGO}" go get github.com/onsi/ginkgo/ginkgo || die "building ginkgo failed" && + mv "$GOPATH/bin/ginkgo" "${CSI_PROW_BIN}" +} + +# This checks out a repo ("https://github.com/kubernetes/kubernetes") +# in a certain location ("$GOPATH/src/k8s.io/kubernetes") at +# a certain revision (a hex commit hash, v1.13.1, master). It's okay +# for that directory to exist already. +git_checkout () { + local repo path revision + repo="$1" + shift + path="$1" + shift + revision="$1" + shift + + mkdir -p "$path" + if ! [ -d "$path/.git" ]; then + run git init "$path" + fi + if (cd "$path" && run git fetch "$@" "$repo" "$revision"); then + (cd "$path" && run git checkout FETCH_HEAD) || die "checking out $repo $revision failed" + else + # Might have been because fetching by revision is not + # supported by GitHub (https://github.com/isaacs/github/issues/436). + # Fall back to fetching everything. + (cd "$path" && run git fetch "$repo" '+refs/heads/*:refs/remotes/csiprow/heads/*' '+refs/tags/*:refs/tags/*') || die "fetching $repo failed" + (cd "$path" && run git checkout "$revision") || die "checking out $repo $revision failed" + fi +} + +list_gates () ( + set -f; IFS=',' + # Ignore: Double quote to prevent globbing and word splitting. + # shellcheck disable=SC2086 + set -- $1 + while [ "$1" ]; do + # Ignore: See if you can use ${variable//search/replace} instead. + # shellcheck disable=SC2001 + echo "$1" | sed -e 's/ *\([^ =]*\) *= *\([^ ]*\) */ \1: \2/' + shift + done +) + +csi_prow_kind_have_kubernetes=false +# Brings up a Kubernetes cluster and sets KUBECONFIG. +# Accepts additional feature gates in the form gate1=true|false,gate2=... +start_cluster () { + local image gates + gates="$1" + + if kind get clusters | grep -q csi-prow; then + run kind delete cluster --name=csi-prow || die "kind delete failed" + fi + + # Build from source? + if [[ "${CSI_PROW_KUBERNETES_VERSION}" =~ ^release-|^latest$ ]]; then + if ! ${csi_prow_kind_have_kubernetes}; then + local version="${CSI_PROW_KUBERNETES_VERSION}" + if [ "$version" = "latest" ]; then + version=master + fi + git_checkout https://github.com/kubernetes/kubernetes "$GOPATH/src/k8s.io/kubernetes" "$version" --depth=1 || die "checking out Kubernetes $version failed" + + # "kind build" and/or the Kubernetes build rules need at least one tag, which we don't have + # when doing a shallow fetch. Therefore we fake one. For some reason, v1.0.0 didn't work + # while v1.14.0-fake.1 did. + (cd "$GOPATH/src/k8s.io/kubernetes" && run git tag -f v1.14.0-fake.1) || die "git tag failed" + + run_with_go "${CSI_PROW_GO_VERSION_K8S}" kind build node-image --type bazel --image csiprow/node:latest --kube-root "$GOPATH/src/k8s.io/kubernetes" || die "'kind build node-image' failed" + csi_prow_kind_have_kubernetes=true + fi + image="csiprow/node:latest" + else + image="kindest/node:v${CSI_PROW_KUBERNETES_VERSION}" + fi + cat >"${CSI_PROW_WORK}/kind-config.yaml" </dev/null; wait) + info "For container output see job artifacts." + die "deploying the hostpath driver with ${deploy_hostpath} failed" + fi +} + +# collect logs and cluster status (like the version of all components, Kubernetes version, test version) +collect_cluster_info () { + cat <>"${ARTIFACTS}/$pod/$container.log" & + echo "$!" + done + done +} + +# Makes the E2E test suite binary available as "${CSI_PROW_WORK}/e2e.test". +install_e2e () { + if [ -e "${CSI_PROW_WORK}/e2e.test" ]; then + return + fi + + git_checkout "${CSI_PROW_E2E_REPO}" "${GOPATH}/src/${CSI_PROW_E2E_IMPORT_PATH}" "${CSI_PROW_E2E_VERSION}" --depth=1 && + if [ "${CSI_PROW_E2E_IMPORT_PATH}" = "k8s.io/kubernetes" ]; then + run_with_go "${CSI_PROW_GO_VERSION_E2E}" make WHAT=test/e2e/e2e.test "-C${GOPATH}/src/${CSI_PROW_E2E_IMPORT_PATH}" && + ln -s "${GOPATH}/src/${CSI_PROW_E2E_IMPORT_PATH}/_output/bin/e2e.test" "${CSI_PROW_WORK}" + else + run_with_go "${CSI_PROW_GO_VERSION_E2E}" go test -c -o "${CSI_PROW_WORK}/e2e.test" "${CSI_PROW_E2E_IMPORT_PATH}/test/e2e" + fi +} + +# Makes the csi-sanity test suite binary available as +# "${CSI_PROW_WORK}/csi-sanity". +install_sanity () ( + if [ -e "${CSI_PROW_WORK}/csi-sanity" ]; then + return + fi + + git_checkout "${CSI_PROW_SANITY_REPO}" "${GOPATH}/src/${CSI_PROW_SANITY_IMPORT_PATH}" "${CSI_PROW_SANITY_VERSION}" --depth=1 || die "checking out csi-sanity failed" + run_with_go "${CSI_PROW_GO_VERSION_SANITY}" go test -c -o "${CSI_PROW_WORK}/csi-sanity" "${CSI_PROW_SANITY_IMPORT_PATH}/cmd/csi-sanity" || die "building csi-sanity failed" +) + +# Whether the hostpath driver supports raw block devices depends on which version +# we are testing. It would be much nicer if we could determine that by querying the +# installed driver. +hostpath_supports_block () { + if [ -e "cmd/hostpathplugin" ] && ${CSI_PROW_BUILD_JOB}; then + # The assumption is that if we build the hostpath driver, then it is + # a current version with support. + echo true + return + fi + + case "${CSI_PROW_DEPLOYMENT}" in kubernetes-1.13) echo false;; # wasn't supported and probably won't be backported + *) echo true;; # probably all other deployments have a recent driver + esac +} + +# Captures pod output while running some other command. +run_with_loggers () ( + loggers=$(start_loggers -f) + trap 'kill $loggers' EXIT + + run "$@" +) + +# Invokes the filter-junit.go tool. +run_filter_junit () { + run_with_go "${CSI_PROW_GO_VERSION_BUILD}" go run "${RELEASE_TOOLS_ROOT}/filter-junit.go" "$@" +} + +# Runs the E2E test suite in a sub-shell. +run_e2e () ( + name="$1" + shift + + install_e2e || die "building e2e.test failed" + install_ginkgo || die "installing ginkgo failed" + + # TODO (?): multi-node cluster (depends on https://github.com/kubernetes-csi/csi-driver-host-path/pull/14). + # When running on a multi-node cluster, we need to figure out where the + # hostpath driver was deployed and set ClientNodeName accordingly. + + # The content of this file depends on both what the E2E suite expects and + # what the installed hostpath driver supports. Generating it here seems + # prone to breakage, but it is uncertain where a better place might be. + cat >"${CSI_PROW_WORK}/hostpath-test-driver.yaml" </dev/null >/dev/null; then + run_filter_junit -t="External Storage" -o "${ARTIFACTS}/junit_${name}.xml" "${ARTIFACTS}"/junit_[0-9]*.xml && rm -f "${ARTIFACTS}"/junit_[0-9]*.xml + fi + } + trap move_junit EXIT + + cd "${GOPATH}/src/${CSI_PROW_E2E_IMPORT_PATH}" && + run_with_loggers ginkgo -v "$@" "${CSI_PROW_WORK}/e2e.test" -- -report-dir "${ARTIFACTS}" -storage.testdriver="${CSI_PROW_WORK}/hostpath-test-driver.yaml" +) + +# Run csi-sanity against installed CSI driver. +run_sanity () ( + install_sanity || die "installing csi-sanity failed" + + cat >"${CSI_PROW_WORK}/mkdir_in_pod.sh" <"${CSI_PROW_WORK}/rmdir_in_pod.sh" </\>/g' -e 's/\x1B...//g' +} + +# The "make test" output starts each test with "### :" +# and then ends when the next test starts or with "make: *** +# [] Error 1" when there was a failure. Here we read each +# line of that output, split it up into individual tests and generate +# a make-test.xml file in JUnit format. +make_test_to_junit () { + local ret out testname testoutput + ret=0 + # Plain make-test.xml was not delivered as text/xml by the web + # server and ignored by spyglass. It seems that the name has to + # match junit*.xml. + out="${ARTIFACTS}/junit_make_test.xml" + testname= + echo "" >>"$out" + + while IFS= read -r line; do + echo "$line" # pass through + if echo "$line" | grep -q "^### [^ ]*:$"; then + if [ "$testname" ]; then + # previous test succesful + echo " " >>"$out" + echo " " >>"$out" + fi + # Ignore: See if you can use ${variable//search/replace} instead. + # shellcheck disable=SC2001 + # + # start new test + testname="$(echo "$line" | sed -e 's/^### \([^ ]*\):$/\1/')" + testoutput= + echo " " >>"$out" + echo " " >>"$out" + elif echo "$line" | grep -q '^make: .*Error [0-9]*$'; then + if [ "$testname" ]; then + # Ignore: Consider using { cmd1; cmd2; } >> file instead of individual redirects. + # shellcheck disable=SC2129 + # + # end test with failure + echo " " >>"$out" + # Include the same text as in also in , + # because then it is easier to view in spyglass (shown directly + # instead of having to click through to stdout). + echo " " >>"$out" + echo -n "$testoutput" | ascii_to_xml >>"$out" + echo " " >>"$out" + echo " " >>"$out" + fi + # remember failure for exit code + ret=1 + # not currently inside a test + testname= + else + if [ "$testname" ]; then + # Test output. + echo "$line" | ascii_to_xml >>"$out" + testoutput="$testoutput$line +" + fi + fi + done + # if still in a test, close it now + if [ "$testname" ]; then + echo " " >>"$out" + echo " " >>"$out" + fi + echo "" >>"$out" + + # this makes the error more visible in spyglass + if [ "$ret" -ne 0 ]; then + echo "ERROR: 'make test' failed" + return 1 + fi +} + +main () { + local images ret + ret=0 + + images= + if ${CSI_PROW_BUILD_JOB}; then + # A successful build is required for testing. + run_with_go "${CSI_PROW_GO_VERSION_BUILD}" make all || die "'make all' failed" + # We don't want test failures to prevent E2E testing below, because the failure + # might have been minor or unavoidable, for example when experimenting with + # changes in "release-tools" in a PR (that fails the "is release-tools unmodified" + # test). + if test_enabled "unit"; then + if ! run_with_go "${CSI_PROW_GO_VERSION_BUILD}" make -k test 2>&1 | make_test_to_junit; then + warn "'make test' failed, proceeding anyway" + ret=1 + fi + fi + # Required for E2E testing. + run_with_go "${CSI_PROW_GO_VERSION_BUILD}" make container || die "'make container' failed" + fi + + install_kind || die "installing kind failed" + start_cluster || die "starting the cluster failed" + + if ${CSI_PROW_BUILD_JOB}; then + cmds="$(grep '^\s*CMDS\s*=' Makefile | sed -e 's/\s*CMDS\s*=//')" + # Get the image that was just built (if any) from the + # top-level Makefile CMDS variable and set the + # deploy-hostpath.sh env variables for it. We also need to + # side-load those images into the cluster. + for i in $cmds; do + e=$(echo "$i" | tr '[:lower:]' '[:upper:]' | tr - _) + images="$images ${e}_REGISTRY=none ${e}_TAG=csiprow" + + # We must avoid the tag "latest" because that implies + # always pulling the image + # (https://github.com/kubernetes-sigs/kind/issues/328). + docker tag "$i:latest" "$i:csiprow" || die "tagging the locally built container image for $i failed" + kind load docker-image --name csi-prow "$i:csiprow" || die "could not load the $i:latest image into the kind cluster" + done + + if [ -e deploy/kubernetes/rbac.yaml ]; then + # This is one of those components which has its own RBAC rules (like external-provisioner). + # We are testing a locally built image and also want to test with the the current, + # potentially modified RBAC rules. + if [ "$(echo "$cmds" | wc -w)" != 1 ]; then + die "ambiguous deploy/kubernetes/rbac.yaml: need exactly one command, got: $cmds" + fi + e=$(echo "$cmds" | tr '[:lower:]' '[:upper:]' | tr - _) + images="$images ${e}_RBAC=$(pwd)/deploy/kubernetes/rbac.yaml" + fi + fi + + # Installing the driver might be disabled, in which case we bail out early. + if ! install_hostpath "$images"; then + info "hostpath driver installation disabled, skipping E2E testing" + return "$ret" + fi + + collect_cluster_info + + if test_enabled "sanity"; then + if ! run_sanity; then + ret=1 + fi + fi + + if test_enabled "parallel"; then + # Ignore: Double quote to prevent globbing and word splitting. + # shellcheck disable=SC2086 + if ! run_e2e parallel ${CSI_PROW_GINKO_PARALLEL} \ + -focus="External.Storage" \ + -skip="$(regex_join "${CSI_PROW_E2E_SERIAL}" "${CSI_PROW_E2E_ALPHA}" "${CSI_PROW_E2E_SKIP}")"; then + warn "E2E parallel failed" + ret=1 + fi + fi + + if test_enabled "serial"; then + if ! run_e2e serial \ + -focus="External.Storage.*($(regex_join "${CSI_PROW_E2E_SERIAL}"))" \ + -skip="$(regex_join "${CSI_PROW_E2E_ALPHA}" "${CSI_PROW_E2E_SKIP}")"; then + warn "E2E serial failed" + ret=1 + fi + fi + + if (test_enabled "parallel-alpha" || test_enabled "serial-alpha") && [ "${CSI_PROW_E2E_ALPHA_GATES}" ]; then + # Need to (re)create the cluster. + start_cluster "${CSI_PROW_E2E_ALPHA_GATES}" || die "starting alpha cluster failed" + if ${CSI_PROW_BUILD_JOB}; then + # Ignore: Double quote to prevent globbing and word splitting. + # Ignore: To read lines rather than words, pipe/redirect to a 'while read' loop. + # shellcheck disable=SC2086 disable=SC2013 + for i in $(grep '^\s*CMDS\s*=' Makefile | sed -e 's/\s*CMDS\s*=//'); do + kind load docker-image --name csi-prow $i:csiprow || die "could not load the $i:latest image into the kind cluster" + done + fi + install_hostpath "$images" || die "hostpath driver installation failed unexpectedly on alpha cluster" + + if test_enabled "parallel-alpha"; then + # Ignore: Double quote to prevent globbing and word splitting. + # shellcheck disable=SC2086 + if ! run_e2e parallel-alpha ${CSI_PROW_GINKO_PARALLEL} \ + -focus="External.Storage.*($(regex_join "${CSI_PROW_E2E_ALPHA}"))" \ + -skip="$(regex_join "${CSI_PROW_E2E_SERIAL}" "${CSI_PROW_E2E_SKIP}")"; then + warn "E2E parallel alpha failed" + ret=1 + fi + fi + + if test_enabled "serial-alpha"; then + if ! run_e2e serial-alpha \ + -focus="External.Storage.*(($(regex_join "${CSI_PROW_E2E_SERIAL}")).*($(regex_join "${CSI_PROW_E2E_ALPHA}"))|($(regex_join "${CSI_PROW_E2E_ALPHA}")).*($(regex_join "${CSI_PROW_E2E_SERIAL}")))" \ + -skip="$(regex_join "${CSI_PROW_E2E_SKIP}")"; then + warn "E2E serial alpha failed" + ret=1 + fi + fi + fi + + # Merge all junit files into one. This gets rid of duplicated "skipped" tests. + if ls "${ARTIFACTS}"/junit_*.xml 2>/dev/null >&2; then + run_filter_junit -o "${CSI_PROW_WORK}/junit_final.xml" "${ARTIFACTS}"/junit_*.xml && rm "${ARTIFACTS}"/junit_*.xml && mv "${CSI_PROW_WORK}/junit_final.xml" "${ARTIFACTS}" + fi + + return "$ret" +} From 0a0fd49b8b146c21e9c90ffbfd696b83b6ed86b7 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Tue, 2 Apr 2019 20:53:08 +0200 Subject: [PATCH 023/107] prow.sh: comment clarification --- prow.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prow.sh b/prow.sh index 17e8ae4..faa5c36 100755 --- a/prow.sh +++ b/prow.sh @@ -15,7 +15,7 @@ # limitations under the License. -# This script runs inside a Prow job. It runs unit tests ("make test") +# This script runs inside a Prow job. It can run unit tests ("make test") # and E2E testing. This E2E testing covers different scenarios (see # https://github.com/kubernetes/enhancements/pull/807): # - running the stable hostpath example against a Kubernetes release From 429581c52d1cdbd1d1c3f9c25411fe6aa8c66610 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Tue, 2 Apr 2019 20:53:12 +0200 Subject: [PATCH 024/107] prow.sh: pull Go version from travis.yml The travis.yml is now the only place where the Go version for the component itself needs to be configured. --- prow.sh | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/prow.sh b/prow.sh index faa5c36..bab7eb0 100755 --- a/prow.sh +++ b/prow.sh @@ -39,6 +39,9 @@ # - kind (https://github.com/kubernetes-sigs/kind) installed # - optional: Go already installed +RELEASE_TOOLS_ROOT="$(realpath "$(dirname "${BASH_SOURCE[0]}")")" +REPO_DIR="$(pwd)" + # Sets the default value for a variable if not set already and logs the value. # Any variable set this way is usually something that a repo's .prow.sh # or the job can set. @@ -53,7 +56,10 @@ configvar () { # If the pre-installed Go is missing or a different # version, the required version here will get installed # from https://golang.org/dl/. -configvar CSI_PROW_GO_VERSION_BUILD 1.11.4 "Go version for building the component" # depends on component's source code +go_from_travis_yml () { + grep "^ *- go:" "${RELEASE_TOOLS_ROOT}/travis.yml" | sed -e 's/.*go: *//' +} +configvar CSI_PROW_GO_VERSION_BUILD "$(go_from_travis_yml)" "Go version for building the component" # depends on component's source code configvar CSI_PROW_GO_VERSION_K8S 1.12.1 "Go version for building Kubernetes for the test cluster" # depends on Kubernetes version configvar CSI_PROW_GO_VERSION_E2E 1.12.1 "Go version for building the Kubernetes E2E test suite" # depends on CSI_PROW_E2E settings below configvar CSI_PROW_GO_VERSION_SANITY "${CSI_PROW_GO_VERSION_BUILD}" "Go version for building the csi-sanity test suite" # depends on CSI_PROW_SANITY settings below @@ -262,9 +268,6 @@ configvar CSI_PROW_E2E_SKIP 'while.kubelet.is.down.*Disruptive' "tests that need configvar ARTIFACTS "${CSI_PROW_WORK}/artifacts" "artifacts" mkdir -p "${ARTIFACTS}" -RELEASE_TOOLS_ROOT="$(realpath "$(dirname "${BASH_SOURCE[0]}")")" -REPO_DIR="$(pwd)" - run () { echo "$(date) $(go version | sed -e 's/.*version \(go[^ ]*\).*/\1/') $(if [ "$(pwd)" != "${REPO_DIR}" ]; then pwd; fi)\$" "$@" >&2 "$@" From 29545bb012b6cbeab74ffbced61951e9b36137d2 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Wed, 3 Apr 2019 12:38:33 +0200 Subject: [PATCH 025/107] prow.sh: take Go version from Kubernetes source Using the same (recent) Go version for all Kubernetes versions can break for older versions when there are incompatible changes in Go. To avoid that, we use exactly the minimum version of Go required for each Kubernetes version. This is based on the assumption that this combination was tested successfully. When building the E2E suite from Kubernetes (the default) we do the same, but still allow building it from elsewhere. We allow the Go version to be empty when it doesn't matter. --- prow.sh | 27 ++++++++++++++++++++++----- 1 file changed, 22 insertions(+), 5 deletions(-) diff --git a/prow.sh b/prow.sh index bab7eb0..6fd55cc 100755 --- a/prow.sh +++ b/prow.sh @@ -60,8 +60,7 @@ go_from_travis_yml () { grep "^ *- go:" "${RELEASE_TOOLS_ROOT}/travis.yml" | sed -e 's/.*go: *//' } configvar CSI_PROW_GO_VERSION_BUILD "$(go_from_travis_yml)" "Go version for building the component" # depends on component's source code -configvar CSI_PROW_GO_VERSION_K8S 1.12.1 "Go version for building Kubernetes for the test cluster" # depends on Kubernetes version -configvar CSI_PROW_GO_VERSION_E2E 1.12.1 "Go version for building the Kubernetes E2E test suite" # depends on CSI_PROW_E2E settings below +configvar CSI_PROW_GO_VERSION_E2E "" "override Go version for building the Kubernetes E2E test suite" # normally doesn't need to be set, see install_e2e configvar CSI_PROW_GO_VERSION_SANITY "${CSI_PROW_GO_VERSION_BUILD}" "Go version for building the csi-sanity test suite" # depends on CSI_PROW_SANITY settings below configvar CSI_PROW_GO_VERSION_KIND "${CSI_PROW_GO_VERSION_BUILD}" "Go version for building 'kind'" # depends on CSI_PROW_KIND_VERSION below configvar CSI_PROW_GO_VERSION_GINKGO "${CSI_PROW_GO_VERSION_BUILD}" "Go version for building ginkgo" # depends on CSI_PROW_GINKGO_VERSION below @@ -292,12 +291,14 @@ mkdir -p "${CSI_PROW_BIN}" PATH="${CSI_PROW_BIN}:$PATH" # Ensure that PATH has the desired version of the Go tools, then run command given as argument. +# Empty parameter uses the already installed Go. In Prow, that version is kept up-to-date by +# bumping the container image regularly. run_with_go () { local version version="$1" shift - if go version 2>/dev/null | grep -q "go$version"; then + if ! [ "$version" ] || go version 2>/dev/null | grep -q "go$version"; then run "$@" else if ! [ -d "${CSI_PROW_WORK}/go-$version" ]; then @@ -374,6 +375,20 @@ list_gates () ( done ) +go_version_for_kubernetes () ( + local path="$1" + local version="$2" + local go_version + + # We use the minimal Go version specified for each K8S release (= minimum_go_version in hack/lib/golang.sh). + # More recent versions might also work, but we don't want to count on that. + go_version="$(grep minimum_go_version= "$path/hack/lib/golang.sh" | sed -e 's/.*=go//')" + if ! [ "$go_version" ]; then + die "Unable to determine Go version for Kubernetes $version from hack/lib/golang.sh." + fi + echo "$go_version" +) + csi_prow_kind_have_kubernetes=false # Brings up a Kubernetes cluster and sets KUBECONFIG. # Accepts additional feature gates in the form gate1=true|false,gate2=... @@ -399,7 +414,8 @@ start_cluster () { # while v1.14.0-fake.1 did. (cd "$GOPATH/src/k8s.io/kubernetes" && run git tag -f v1.14.0-fake.1) || die "git tag failed" - run_with_go "${CSI_PROW_GO_VERSION_K8S}" kind build node-image --type bazel --image csiprow/node:latest --kube-root "$GOPATH/src/k8s.io/kubernetes" || die "'kind build node-image' failed" + go_version="$(go_version_for_kubernetes "$GOPATH/src/k8s.io/kubernetes" "$version")" || die "cannot proceed without knowing Go version for Kubernetes" + run_with_go "$go_version" kind build node-image --type bazel --image csiprow/node:latest --kube-root "$GOPATH/src/k8s.io/kubernetes" || die "'kind build node-image' failed" csi_prow_kind_have_kubernetes=true fi image="csiprow/node:latest" @@ -563,7 +579,8 @@ install_e2e () { git_checkout "${CSI_PROW_E2E_REPO}" "${GOPATH}/src/${CSI_PROW_E2E_IMPORT_PATH}" "${CSI_PROW_E2E_VERSION}" --depth=1 && if [ "${CSI_PROW_E2E_IMPORT_PATH}" = "k8s.io/kubernetes" ]; then - run_with_go "${CSI_PROW_GO_VERSION_E2E}" make WHAT=test/e2e/e2e.test "-C${GOPATH}/src/${CSI_PROW_E2E_IMPORT_PATH}" && + go_version="${CSI_PROW_GO_VERSION_E2E:-$(go_version_for_kubernetes "${GOPATH}/src/${CSI_PROW_E2E_IMPORT_PATH}" "${CSI_PROW_E2E_VERSION}")}" && + run_with_go "$go_version" make WHAT=test/e2e/e2e.test "-C${GOPATH}/src/${CSI_PROW_E2E_IMPORT_PATH}" && ln -s "${GOPATH}/src/${CSI_PROW_E2E_IMPORT_PATH}/_output/bin/e2e.test" "${CSI_PROW_WORK}" else run_with_go "${CSI_PROW_GO_VERSION_E2E}" go test -c -o "${CSI_PROW_WORK}/e2e.test" "${CSI_PROW_E2E_IMPORT_PATH}/test/e2e" From 741319bd308c10b6c7b16c3cce71ef941cb7086b Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Wed, 3 Apr 2019 12:42:33 +0200 Subject: [PATCH 026/107] prow.sh: improve building Kubernetes from source While switching back and forth between release-1.13 and release-1.14 locally, there was the problem that the local kind image kept using the wrong kubelet binary despite rebuilding from source. The problem went away after cleaning the Bazel cache. Exact root cause unknown, but perhaps using unique tags and properly cleaning the repo helps. If not, then the unique tags at least will show what the problem is. --- prow.sh | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/prow.sh b/prow.sh index 6fd55cc..a7ea66f 100755 --- a/prow.sh +++ b/prow.sh @@ -360,6 +360,9 @@ git_checkout () { (cd "$path" && run git fetch "$repo" '+refs/heads/*:refs/remotes/csiprow/heads/*' '+refs/tags/*:refs/tags/*') || die "fetching $repo failed" (cd "$path" && run git checkout "$revision") || die "checking out $repo $revision failed" fi + # This is useful for local testing or when switching between different revisions in the same + # repo. + (cd "$path" && run git clean -fdx) || die "failed to clean $path" } list_gates () ( @@ -410,10 +413,20 @@ start_cluster () { git_checkout https://github.com/kubernetes/kubernetes "$GOPATH/src/k8s.io/kubernetes" "$version" --depth=1 || die "checking out Kubernetes $version failed" # "kind build" and/or the Kubernetes build rules need at least one tag, which we don't have - # when doing a shallow fetch. Therefore we fake one. For some reason, v1.0.0 didn't work - # while v1.14.0-fake.1 did. - (cd "$GOPATH/src/k8s.io/kubernetes" && run git tag -f v1.14.0-fake.1) || die "git tag failed" - + # when doing a shallow fetch. Therefore we fake one: + # release-1.12 -> v1.12.0-release..csiprow + # latest or -> v1.14.0-.csiprow + case "${CSI_PROW_KUBERNETES_VERSION}" in + release-*) + # Ignore: See if you can use ${variable//search/replace} instead. + # shellcheck disable=SC2001 + tag="$(echo "${CSI_PROW_KUBERNETES_VERSION}" | sed -e 's/release-\(.*\)/v\1.0-release./')";; + *) + # We have to make something up. v1.0.0 did not work for some reasons. + tag="v1.14.0-";; + esac + tag="$tag$(cd "$GOPATH/src/k8s.io/kubernetes" && git rev-list --abbrev-commit HEAD).csiprow" + (cd "$GOPATH/src/k8s.io/kubernetes" && run git tag -f "$tag") || die "git tag failed" go_version="$(go_version_for_kubernetes "$GOPATH/src/k8s.io/kubernetes" "$version")" || die "cannot proceed without knowing Go version for Kubernetes" run_with_go "$go_version" kind build node-image --type bazel --image csiprow/node:latest --kube-root "$GOPATH/src/k8s.io/kubernetes" || die "'kind build node-image' failed" csi_prow_kind_have_kubernetes=true From 6602d38bfbce6a526f9225c39aa6b86adbd86978 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Wed, 3 Apr 2019 13:54:01 +0200 Subject: [PATCH 027/107] prow.sh: different E2E suite depending on Kubernetes version Instead of always using the latest E2E tests for all Kubernetes versions, the plan now is to use the tests that match the Kubernetes version. However, for 1.13 we have to make an exception because the suite for that version did not support the necessary --storage.testdriver parameter. --- prow.sh | 72 ++++++++++++++++++++++++++++----------------------------- 1 file changed, 36 insertions(+), 36 deletions(-) diff --git a/prow.sh b/prow.sh index a7ea66f..c54a81b 100755 --- a/prow.sh +++ b/prow.sh @@ -52,6 +52,19 @@ configvar () { eval echo "\$3:" "$1=\${$1}" } +# Prints the value of a variable + version suffix, falling back to variable + "LATEST". +get_versioned_variable () { + local var="$1" + local version="$2" + local value + + eval value="\${${var}_${version}}" + if ! [ "$value" ]; then + eval value="\${${var}_LATEST}" + fi + echo "$value" +} + # Go versions can be specified seperately for different tasks # If the pre-installed Go is missing or a different # version, the required version here will get installed @@ -158,9 +171,15 @@ configvar CSI_PROW_HOSTPATH_CANARY "" "hostpath image" # all generated files are present. # # CSI_PROW_E2E_REPO=none disables E2E testing. -configvar CSI_PROW_E2E_VERSION v1.14.0 "E2E version" -configvar CSI_PROW_E2E_REPO https://github.com/kubernetes/kubernetes "E2E repo" -configvar CSI_PROW_E2E_IMPORT_PATH k8s.io/kubernetes "E2E package" +configvar CSI_PROW_E2E_VERSION_1_13 v1.14.0 "E2E version for Kubernetes 1.13.x" # we can't use the one from 1.13.x because it didn't have --storage.testdriver +configvar CSI_PROW_E2E_VERSION_1_14 v1.14.0 "E2E version for Kubernetes 1.14.x" +# TODO: add new CSI_PROW_E2E_VERSION entry for future Kubernetes releases +configvar CSI_PROW_E2E_VERSION_LATEST master "E2E version for Kubernetes master" # testing against Kubernetes master is already tracking a moving target, so we might as well use a moving E2E version +configvar CSI_PROW_E2E_REPO_LATEST https://github.com/kubernetes/kubernetes "E2E repo for Kubernetes >= 1.13.x" # currently the same for all versions +configvar CSI_PROW_E2E_IMPORT_PATH_LATEST k8s.io/kubernetes "E2E package for Kubernetes >= 1.13.x" # currently the same for all versions +configvar CSI_PROW_E2E_VERSION "$(get_versioned_variable CSI_PROW_E2E_VERSION "${csi_prow_kubernetes_version_suffix}")" "E2E version" +configvar CSI_PROW_E2E_REPO "$(get_versioned_variable CSI_PROW_E2E_REPO "${csi_prow_kubernetes_version_suffix}")" "E2E repo" +configvar CSI_PROW_E2E_IMPORT_PATH "$(get_versioned_variable CSI_PROW_E2E_IMPORT_PATH "${csi_prow_kubernetes_version_suffix}")" "E2E package" # csi-sanity testing from the csi-test repo can be run against the installed # CSI driver. For this to work, deploying the driver must expose the Unix domain @@ -200,42 +219,22 @@ regex_join () { echo "$@" | sed -e 's/ */|/g' -e 's/^|*//' -e 's/|*$//' -e 's/^$/this-matches-nothing/g' } -# Prints the value of a variable + version suffix, falling back to variable + "LATEST". -get_versioned_variable () { - local var="$1" - local version="$2" - local value - - eval value="\${${var}_${version}}" - if ! [ "$value" ]; then - eval value="\${${var}_LATEST}" - fi - echo "$value" -} - -# Which tests are alpha depends on the Kubernetes version. The same -# E2E test suite is used for all Kubernetes versions, including older -# Kubernetes. -# -# Feature tags in the test are set based on what is an alpha -# feature in the Kubernetes version that contains the E2E test suite's -# source code. +# Which tests are alpha depends on the Kubernetes version. We could +# use the same E2E test for all Kubernetes version. This would have +# the advantage that new tests can be applied to older versions +# without having to backport tests. # -# So when testing against an older Kubernetes release, some tests -# might be enabled by default which don't pass for that older -# Kubernetes version. In that case, the -# CSI_PROW_E2E_ALPHA_ variable cannot just -# be based on the `Feature` tag, it also must contain the names -# of tests that used to be alpha but no longer have that tag. +# But the feature tag gets removed from E2E tests when the corresponding +# feature becomes beta, so we would have to track which tests were +# alpha in previous Kubernetes releases. This was considered too +# error prone. Therefore we use E2E tests that match the Kubernetes +# version that is getting tested. # -# is just major+minor version separated by -# underscore. "latest" matches master. It is also used for -# unknown versions, so when master gets released there is -# no need to add a new versioned variable. That only needs to be -# done when updating something in the configuration that leads -# to "latest" no longer being suitable. +# However, for 1.13.x testing we have to use the E2E tests from 1.14 +# because 1.13 didn't have --storage.testdriver yet, so for that (and only +# that version) we have to define alpha tests differently. configvar CSI_PROW_E2E_ALPHA_1_13 '\[Feature: \[Testpattern:.Dynamic.PV..block.volmode.\] should.create.and.delete.block.persistent.volumes' "alpha tests for Kubernetes 1.13" # Raw block was an alpha feature in 1.13. -configvar CSI_PROW_E2E_ALPHA_LATEST '\[Feature:' "alpha tests for Kubernetes master" +configvar CSI_PROW_E2E_ALPHA_LATEST '\[Feature:' "alpha tests for Kubernetes >= 1.14" # there's no need to update this, adding a new case for CSI_PROW_E2E for a new Kubernetes is enough configvar CSI_PROW_E2E_ALPHA "$(get_versioned_variable CSI_PROW_E2E_ALPHA "${csi_prow_kubernetes_version_suffix}")" "alpha tests" # After the parallel E2E test without alpha features, a test cluster @@ -251,6 +250,7 @@ configvar CSI_PROW_E2E_ALPHA "$(get_versioned_variable CSI_PROW_E2E_ALPHA "${csi # the failing test for "latest" or by updating the test and not running # it anymore for older releases. configvar CSI_PROW_E2E_ALPHA_GATES_1_13 'VolumeSnapshotDataSource=true,BlockVolume=true,CSIBlockVolume=true' "alpha feature gates for Kubernetes 1.13" +# TODO: add new CSI_PROW_ALPHA_GATES entry for future Kubernetes releases configvar CSI_PROW_E2E_ALPHA_GATES_LATEST 'VolumeSnapshotDataSource=true' "alpha feature gates for latest Kubernetes" configvar CSI_PROW_E2E_ALPHA_GATES "$(get_versioned_variable CSI_PROW_E2E_ALPHA_GATES "${csi_prow_kubernetes_version_suffix}")" "alpha E2E feature gates" From d87eccb46dca47496f7f03bce1909e9ba548f00e Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Wed, 3 Apr 2019 14:59:33 +0200 Subject: [PATCH 028/107] prow.sh: switch back to upstream csi-driver-host-path The temporary fork was merged, we can use the upstream repo again. --- prow.sh | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/prow.sh b/prow.sh index c54a81b..490fe99 100755 --- a/prow.sh +++ b/prow.sh @@ -151,12 +151,8 @@ configvar CSI_PROW_WORK "$(mkdir -p "$GOPATH/pkg" && mktemp -d "$GOPATH/pkg/csip # # When no deploy script is found (nothing in `deploy` directory, # CSI_PROW_HOSTPATH_REPO=none), nothing gets deployed. -# -# TODO: merge https://github.com/kubernetes-csi/csi-driver-host-path/pull/29, switch to revision from master -configvar CSI_PROW_HOSTPATH_VERSION deployment "hostpath driver" -configvar CSI_PROW_HOSTPATH_REPO https://github.com/pohly/csi-driver-host-path "hostpath repo" -# Ignore 'See if you can use ${variable//search/replace} instead.' -# shellcheck disable=SC2001 +configvar CSI_PROW_HOSTPATH_VERSION 486074dc3beef59955faf7bb5210418d9844e0a7 "hostpath driver" # pre-1.1.0 +configvar CSI_PROW_HOSTPATH_REPO https://github.com/kubernetes-csi/csi-driver-host-path "hostpath repo" configvar CSI_PROW_DEPLOYMENT "" "deployment" # If CSI_PROW_HOSTPATH_CANARY is set (typically to "canary", but also From f5014439fc2b977a357ff3b49fd6aab052302f7c Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Fri, 5 Apr 2019 08:44:53 +0200 Subject: [PATCH 029/107] prow.sh: AllAlpha=true for unknown Kubernetes versions This ensures that also new, currently unknown alpha gates are enabled when testing against a future Kubernetes versions. For all currently known Kubernetes versions we just use the minimal set of alpha gates, which ensures that we don't miss any of them in our documentation. --- prow.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/prow.sh b/prow.sh index 490fe99..5a636bc 100755 --- a/prow.sh +++ b/prow.sh @@ -246,8 +246,9 @@ configvar CSI_PROW_E2E_ALPHA "$(get_versioned_variable CSI_PROW_E2E_ALPHA "${csi # the failing test for "latest" or by updating the test and not running # it anymore for older releases. configvar CSI_PROW_E2E_ALPHA_GATES_1_13 'VolumeSnapshotDataSource=true,BlockVolume=true,CSIBlockVolume=true' "alpha feature gates for Kubernetes 1.13" +configvar CSI_PROW_E2E_ALPHA_GATES_1_14 'VolumeSnapshotDataSource=true' "alpha feature gates for Kubernetes 1.14" # TODO: add new CSI_PROW_ALPHA_GATES entry for future Kubernetes releases -configvar CSI_PROW_E2E_ALPHA_GATES_LATEST 'VolumeSnapshotDataSource=true' "alpha feature gates for latest Kubernetes" +configvar CSI_PROW_E2E_ALPHA_GATES_LATEST 'AllAlpha=true' "alpha feature gates for latest Kubernetes" configvar CSI_PROW_E2E_ALPHA_GATES "$(get_versioned_variable CSI_PROW_E2E_ALPHA_GATES "${csi_prow_kubernetes_version_suffix}")" "alpha E2E feature gates" # Some tests are known to be unusable in a KinD cluster. For example, From 31dfaf31dc6fb132255f9696a4562ab3e8a741ae Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Mon, 8 Apr 2019 08:44:10 +0200 Subject: [PATCH 030/107] prow.sh: fix running of just "alpha" tests "grep -w" treated "serial-alpha" as two words and therefore CSI_PROW_TESTS sometimes ran too many tests. --- prow.sh | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/prow.sh b/prow.sh index 5a636bc..f39b339 100755 --- a/prow.sh +++ b/prow.sh @@ -203,7 +203,16 @@ configvar CSI_PROW_SANITY_CONTAINER "hostpath" "Kubernetes container with CSI dr # by setting CSI_PROW_TESTS_SANITY. configvar CSI_PROW_TESTS "unit parallel serial parallel-alpha serial-alpha ${CSI_PROW_TESTS_SANITY}" "tests to run" test_enabled () { - echo "${CSI_PROW_TESTS}" | grep -q -w -e "$1" + local test="$1" + # We want word-splitting here, so ignore: Double quote to prevent globbing and word splitting. + # shellcheck disable=SC2086 + set ${CSI_PROW_TESTS} + for t in "$@"; do + if [ "$t" = "$test" ]; then + return + fi + done + return 1 } # Serial vs. parallel is always determined by these regular expressions. From f3d1d2df5c85dede9cf25201c69634551a538b92 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Mon, 8 Apr 2019 08:47:15 +0200 Subject: [PATCH 031/107] prow.sh: fix hostpath driver version check The previous logic failed for canary jobs, those also deploy a recent driver. Instead of guessing what driver gets installed based on job parameters, check what really runs in the cluster and base the decision on that. We only need to maintain this blacklist for 1.0.x until we replace it with 1.1.0, then this entire hostpath_supports_block can be removed. --- prow.sh | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/prow.sh b/prow.sh index f39b339..cd04131 100755 --- a/prow.sh +++ b/prow.sh @@ -619,18 +619,18 @@ install_sanity () ( # Whether the hostpath driver supports raw block devices depends on which version # we are testing. It would be much nicer if we could determine that by querying the -# installed driver. +# installed driver's capabilities instead of having to do a version check. hostpath_supports_block () { - if [ -e "cmd/hostpathplugin" ] && ${CSI_PROW_BUILD_JOB}; then - # The assumption is that if we build the hostpath driver, then it is - # a current version with support. - echo true - return - fi - - case "${CSI_PROW_DEPLOYMENT}" in kubernetes-1.13) echo false;; # wasn't supported and probably won't be backported - *) echo true;; # probably all other deployments have a recent driver - esac + local result + result="$(docker exec csi-prow-control-plane docker image ls --format='{{.Repository}} {{.Tag}} {{.ID}}' | grep hostpath | while read -r repo tag id; do + if [ "$tag" == "v1.0.1" ]; then + # Old version because the revision label is missing: didn't have support yet. + echo "false" + return + fi + done)" + # If not set, then it must be a newer driver with support. + echo "${result:-true}" } # Captures pod output while running some other command. From aa45a1cd9b9fafde60f7d8a8e314287e3b8a7794 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Mon, 8 Apr 2019 08:51:11 +0200 Subject: [PATCH 032/107] prow.sh: more efficient execution of individual tests When running only some tests, sometimes extra, unnecessarily work was done, like bringing up the cluster without alpha gates. --- prow.sh | 214 ++++++++++++++++++++++++++++++-------------------------- 1 file changed, 116 insertions(+), 98 deletions(-) diff --git a/prow.sh b/prow.sh index cd04131..72be16f 100755 --- a/prow.sh +++ b/prow.sh @@ -202,18 +202,30 @@ configvar CSI_PROW_SANITY_CONTAINER "hostpath" "Kubernetes container with CSI dr # is off by default. A CSI driver can change that default in its .prow.sh # by setting CSI_PROW_TESTS_SANITY. configvar CSI_PROW_TESTS "unit parallel serial parallel-alpha serial-alpha ${CSI_PROW_TESTS_SANITY}" "tests to run" -test_enabled () { - local test="$1" - # We want word-splitting here, so ignore: Double quote to prevent globbing and word splitting. - # shellcheck disable=SC2086 - set ${CSI_PROW_TESTS} - for t in "$@"; do - if [ "$t" = "$test" ]; then - return - fi +tests_enabled () { + local t1 t2 + # We want word-splitting here, so ignore: Quote to prevent word splitting, or split robustly with mapfile or read -a. + # shellcheck disable=SC2206 + local tests=(${CSI_PROW_TESTS}) + for t1 in "$@"; do + for t2 in "${tests[@]}"; do + if [ "$t1" = "$t2" ]; then + return + fi + done done return 1 } +tests_need_kind () { + tests_enabled "sanity" "parallel" "serial" "serial-alpha" "parallel-alpha" +} +tests_need_non_alpha_cluster () { + tests_enabled "sanity" "parallel" "serial" +} +tests_need_alpha_cluster () { + tests_enabled "parallel-alpha" "serial-alpha" +} + # Serial vs. parallel is always determined by these regular expressions. # Individual regular expressions are seperated by spaces for readability @@ -521,6 +533,15 @@ install_hostpath () { return 1 fi + if ${CSI_PROW_BUILD_JOB}; then + # Ignore: Double quote to prevent globbing and word splitting. + # Ignore: To read lines rather than words, pipe/redirect to a 'while read' loop. + # shellcheck disable=SC2086 disable=SC2013 + for i in $(grep '^\s*CMDS\s*=' Makefile | sed -e 's/\s*CMDS\s*=//'); do + kind load docker-image --name csi-prow $i:csiprow || die "could not load the $i:latest image into the kind cluster" + done + fi + if deploy_hostpath="$(find_deployment "$(pwd)/deploy")"; then : elif [ "${CSI_PROW_HOSTPATH_REPO}" = "none" ]; then @@ -836,7 +857,7 @@ main () { # might have been minor or unavoidable, for example when experimenting with # changes in "release-tools" in a PR (that fails the "is release-tools unmodified" # test). - if test_enabled "unit"; then + if tests_enabled "unit"; then if ! run_with_go "${CSI_PROW_GO_VERSION_BUILD}" make -k test 2>&1 | make_test_to_junit; then warn "'make test' failed, proceeding anyway" ret=1 @@ -846,102 +867,99 @@ main () { run_with_go "${CSI_PROW_GO_VERSION_BUILD}" make container || die "'make container' failed" fi - install_kind || die "installing kind failed" - start_cluster || die "starting the cluster failed" + if tests_need_kind; then + install_kind || die "installing kind failed" - if ${CSI_PROW_BUILD_JOB}; then - cmds="$(grep '^\s*CMDS\s*=' Makefile | sed -e 's/\s*CMDS\s*=//')" - # Get the image that was just built (if any) from the - # top-level Makefile CMDS variable and set the - # deploy-hostpath.sh env variables for it. We also need to - # side-load those images into the cluster. - for i in $cmds; do - e=$(echo "$i" | tr '[:lower:]' '[:upper:]' | tr - _) - images="$images ${e}_REGISTRY=none ${e}_TAG=csiprow" - - # We must avoid the tag "latest" because that implies - # always pulling the image - # (https://github.com/kubernetes-sigs/kind/issues/328). - docker tag "$i:latest" "$i:csiprow" || die "tagging the locally built container image for $i failed" - kind load docker-image --name csi-prow "$i:csiprow" || die "could not load the $i:latest image into the kind cluster" - done + if ${CSI_PROW_BUILD_JOB}; then + cmds="$(grep '^\s*CMDS\s*=' Makefile | sed -e 's/\s*CMDS\s*=//')" + # Get the image that was just built (if any) from the + # top-level Makefile CMDS variable and set the + # deploy-hostpath.sh env variables for it. We also need to + # side-load those images into the cluster. + for i in $cmds; do + e=$(echo "$i" | tr '[:lower:]' '[:upper:]' | tr - _) + images="$images ${e}_REGISTRY=none ${e}_TAG=csiprow" + + # We must avoid the tag "latest" because that implies + # always pulling the image + # (https://github.com/kubernetes-sigs/kind/issues/328). + docker tag "$i:latest" "$i:csiprow" || die "tagging the locally built container image for $i failed" + done - if [ -e deploy/kubernetes/rbac.yaml ]; then - # This is one of those components which has its own RBAC rules (like external-provisioner). - # We are testing a locally built image and also want to test with the the current, - # potentially modified RBAC rules. - if [ "$(echo "$cmds" | wc -w)" != 1 ]; then - die "ambiguous deploy/kubernetes/rbac.yaml: need exactly one command, got: $cmds" + if [ -e deploy/kubernetes/rbac.yaml ]; then + # This is one of those components which has its own RBAC rules (like external-provisioner). + # We are testing a locally built image and also want to test with the the current, + # potentially modified RBAC rules. + if [ "$(echo "$cmds" | wc -w)" != 1 ]; then + die "ambiguous deploy/kubernetes/rbac.yaml: need exactly one command, got: $cmds" + fi + e=$(echo "$cmds" | tr '[:lower:]' '[:upper:]' | tr - _) + images="$images ${e}_RBAC=$(pwd)/deploy/kubernetes/rbac.yaml" fi - e=$(echo "$cmds" | tr '[:lower:]' '[:upper:]' | tr - _) - images="$images ${e}_RBAC=$(pwd)/deploy/kubernetes/rbac.yaml" - fi - fi - - # Installing the driver might be disabled, in which case we bail out early. - if ! install_hostpath "$images"; then - info "hostpath driver installation disabled, skipping E2E testing" - return "$ret" - fi - - collect_cluster_info - - if test_enabled "sanity"; then - if ! run_sanity; then - ret=1 - fi - fi - - if test_enabled "parallel"; then - # Ignore: Double quote to prevent globbing and word splitting. - # shellcheck disable=SC2086 - if ! run_e2e parallel ${CSI_PROW_GINKO_PARALLEL} \ - -focus="External.Storage" \ - -skip="$(regex_join "${CSI_PROW_E2E_SERIAL}" "${CSI_PROW_E2E_ALPHA}" "${CSI_PROW_E2E_SKIP}")"; then - warn "E2E parallel failed" - ret=1 fi - fi - - if test_enabled "serial"; then - if ! run_e2e serial \ - -focus="External.Storage.*($(regex_join "${CSI_PROW_E2E_SERIAL}"))" \ - -skip="$(regex_join "${CSI_PROW_E2E_ALPHA}" "${CSI_PROW_E2E_SKIP}")"; then - warn "E2E serial failed" - ret=1 - fi - fi - if (test_enabled "parallel-alpha" || test_enabled "serial-alpha") && [ "${CSI_PROW_E2E_ALPHA_GATES}" ]; then - # Need to (re)create the cluster. - start_cluster "${CSI_PROW_E2E_ALPHA_GATES}" || die "starting alpha cluster failed" - if ${CSI_PROW_BUILD_JOB}; then - # Ignore: Double quote to prevent globbing and word splitting. - # Ignore: To read lines rather than words, pipe/redirect to a 'while read' loop. - # shellcheck disable=SC2086 disable=SC2013 - for i in $(grep '^\s*CMDS\s*=' Makefile | sed -e 's/\s*CMDS\s*=//'); do - kind load docker-image --name csi-prow $i:csiprow || die "could not load the $i:latest image into the kind cluster" - done - fi - install_hostpath "$images" || die "hostpath driver installation failed unexpectedly on alpha cluster" - - if test_enabled "parallel-alpha"; then - # Ignore: Double quote to prevent globbing and word splitting. - # shellcheck disable=SC2086 - if ! run_e2e parallel-alpha ${CSI_PROW_GINKO_PARALLEL} \ - -focus="External.Storage.*($(regex_join "${CSI_PROW_E2E_ALPHA}"))" \ - -skip="$(regex_join "${CSI_PROW_E2E_SERIAL}" "${CSI_PROW_E2E_SKIP}")"; then - warn "E2E parallel alpha failed" - ret=1 + if tests_need_non_alpha_cluster; then + start_cluster || die "starting the non-alpha cluster failed" + + # Installing the driver might be disabled. + if install_hostpath "$images"; then + collect_cluster_info + + if tests_enabled "sanity"; then + if ! run_sanity; then + ret=1 + fi + fi + + if tests_enabled "parallel"; then + # Ignore: Double quote to prevent globbing and word splitting. + # shellcheck disable=SC2086 + if ! run_e2e parallel ${CSI_PROW_GINKO_PARALLEL} \ + -focus="External.Storage" \ + -skip="$(regex_join "${CSI_PROW_E2E_SERIAL}" "${CSI_PROW_E2E_ALPHA}" "${CSI_PROW_E2E_SKIP}")"; then + warn "E2E parallel failed" + ret=1 + fi + fi + + if tests_enabled "serial"; then + if ! run_e2e serial \ + -focus="External.Storage.*($(regex_join "${CSI_PROW_E2E_SERIAL}"))" \ + -skip="$(regex_join "${CSI_PROW_E2E_ALPHA}" "${CSI_PROW_E2E_SKIP}")"; then + warn "E2E serial failed" + ret=1 + fi + fi fi fi - if test_enabled "serial-alpha"; then - if ! run_e2e serial-alpha \ - -focus="External.Storage.*(($(regex_join "${CSI_PROW_E2E_SERIAL}")).*($(regex_join "${CSI_PROW_E2E_ALPHA}"))|($(regex_join "${CSI_PROW_E2E_ALPHA}")).*($(regex_join "${CSI_PROW_E2E_SERIAL}")))" \ - -skip="$(regex_join "${CSI_PROW_E2E_SKIP}")"; then - warn "E2E serial alpha failed" - ret=1 + if tests_need_alpha_cluster && [ "${CSI_PROW_E2E_ALPHA_GATES}" ]; then + # Need to (re)create the cluster. + start_cluster "${CSI_PROW_E2E_ALPHA_GATES}" || die "starting alpha cluster failed" + + # Installing the driver might be disabled. + if install_hostpath "$images"; then + collect_cluster_info + + if tests_enabled "parallel-alpha"; then + # Ignore: Double quote to prevent globbing and word splitting. + # shellcheck disable=SC2086 + if ! run_e2e parallel-alpha ${CSI_PROW_GINKO_PARALLEL} \ + -focus="External.Storage.*($(regex_join "${CSI_PROW_E2E_ALPHA}"))" \ + -skip="$(regex_join "${CSI_PROW_E2E_SERIAL}" "${CSI_PROW_E2E_SKIP}")"; then + warn "E2E parallel alpha failed" + ret=1 + fi + fi + + if tests_enabled "serial-alpha"; then + if ! run_e2e serial-alpha \ + -focus="External.Storage.*(($(regex_join "${CSI_PROW_E2E_SERIAL}")).*($(regex_join "${CSI_PROW_E2E_ALPHA}"))|($(regex_join "${CSI_PROW_E2E_ALPHA}")).*($(regex_join "${CSI_PROW_E2E_SERIAL}")))" \ + -skip="$(regex_join "${CSI_PROW_E2E_SKIP}")"; then + warn "E2E serial alpha failed" + ret=1 + fi + fi fi fi fi From 9b0d9cd74370bff4dfaf86c4ab75973ec9aa56a2 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Mon, 8 Apr 2019 11:37:01 +0200 Subject: [PATCH 033/107] build.make: skip shellcheck if Docker is not available Not all environments have Docker. The simplifying assumption here is that if the Docker command is available, it's also usable. --- build.make | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/build.make b/build.make index 078100c..e5769c5 100644 --- a/build.make +++ b/build.make @@ -141,6 +141,10 @@ test: test-shellcheck test-shellcheck: @ echo; echo "### $@:" @ ret=0; \ + if ! command -v docker; then \ + echo "skipped, no Docker"; \ + return 0; \ + fi; \ for dir in $(abspath $(TEST_SHELLCHECK_DIRS)); do \ echo; \ echo "$$dir:"; \ From 546d5504a1902276561ee1bfc3811afb61346855 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Mon, 8 Apr 2019 19:42:38 +0200 Subject: [PATCH 034/107] prow.sh: debug failing KinD cluster creation When KinD fails in a Prow job, we need additional information to understand why it failed. --- prow.sh | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/prow.sh b/prow.sh index 72be16f..c207568 100755 --- a/prow.sh +++ b/prow.sh @@ -488,7 +488,17 @@ $(list_gates "$gates") featureGates: $(list_gates "$gates") EOF - run kind create cluster --name csi-prow --config "${CSI_PROW_WORK}/kind-config.yaml" --wait 5m --image "$image" || die "'kind create cluster' failed" + info "kind-config.yaml:" + cat "${CSI_PROW_WORK}/kind-config.yaml" + if ! run kind create cluster --name csi-prow --config "${CSI_PROW_WORK}/kind-config.yaml" --wait 5m --image "$image"; then + warn "Cluster creation failed. Will try again with higher verbosity." + info "Available Docker images:" + docker image ls + if ! run kind --loglevel debug create cluster --retain --name csi-prow --config "${CSI_PROW_WORK}/kind-config.yaml" --wait 5m --image "$image"; then + run kind export logs --name csi-prow "$ARTIFACTS/kind-cluster" + die "Cluster creation failed again, giving up. See the 'kind-cluster' artifact directory for additional logs." + fi + fi KUBECONFIG="$(kind get kubeconfig-path --name=csi-prow)" export KUBECONFIG } From cda2fc5874b42ed68b68cbb60e92fff6ab7ba4c0 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Mon, 8 Apr 2019 19:45:44 +0200 Subject: [PATCH 035/107] prow.sh: avoid AllAlpha=true It turned out to not work. Instead of reverting the commit which introduced this, let's better document this explicitly. --- prow.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/prow.sh b/prow.sh index c207568..0a2b37d 100755 --- a/prow.sh +++ b/prow.sh @@ -267,9 +267,9 @@ configvar CSI_PROW_E2E_ALPHA "$(get_versioned_variable CSI_PROW_E2E_ALPHA "${csi # the failing test for "latest" or by updating the test and not running # it anymore for older releases. configvar CSI_PROW_E2E_ALPHA_GATES_1_13 'VolumeSnapshotDataSource=true,BlockVolume=true,CSIBlockVolume=true' "alpha feature gates for Kubernetes 1.13" -configvar CSI_PROW_E2E_ALPHA_GATES_1_14 'VolumeSnapshotDataSource=true' "alpha feature gates for Kubernetes 1.14" +configvar CSI_PROW_E2E_ALPHA_GATES_1_14 'VolumeSnapshotDataSource=true,ExpandCSIVolumes=true' "alpha feature gates for Kubernetes 1.14" # TODO: add new CSI_PROW_ALPHA_GATES entry for future Kubernetes releases -configvar CSI_PROW_E2E_ALPHA_GATES_LATEST 'AllAlpha=true' "alpha feature gates for latest Kubernetes" +configvar CSI_PROW_E2E_ALPHA_GATES_LATEST 'AllAlpha=true,ExpandCSIVolumes=true' "alpha feature gates for latest Kubernetes" configvar CSI_PROW_E2E_ALPHA_GATES "$(get_versioned_variable CSI_PROW_E2E_ALPHA_GATES "${csi_prow_kubernetes_version_suffix}")" "alpha E2E feature gates" # Some tests are known to be unusable in a KinD cluster. For example, From 7aaac225719beb59cf984b885bd8241c699723e7 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Wed, 10 Apr 2019 08:44:54 +0200 Subject: [PATCH 036/107] prow.sh: remove AllAlpha=all, part II This was already meant to be done earlier in cda2fc5874b42ed68b. While at it, extend the permanent TODO with guidance on future feature gates. --- prow.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/prow.sh b/prow.sh index 0a2b37d..57415d4 100755 --- a/prow.sh +++ b/prow.sh @@ -268,8 +268,9 @@ configvar CSI_PROW_E2E_ALPHA "$(get_versioned_variable CSI_PROW_E2E_ALPHA "${csi # it anymore for older releases. configvar CSI_PROW_E2E_ALPHA_GATES_1_13 'VolumeSnapshotDataSource=true,BlockVolume=true,CSIBlockVolume=true' "alpha feature gates for Kubernetes 1.13" configvar CSI_PROW_E2E_ALPHA_GATES_1_14 'VolumeSnapshotDataSource=true,ExpandCSIVolumes=true' "alpha feature gates for Kubernetes 1.14" -# TODO: add new CSI_PROW_ALPHA_GATES entry for future Kubernetes releases -configvar CSI_PROW_E2E_ALPHA_GATES_LATEST 'AllAlpha=true,ExpandCSIVolumes=true' "alpha feature gates for latest Kubernetes" +# TODO: add new CSI_PROW_ALPHA_GATES_xxx entry for future Kubernetes releases and +# add new gates to CSI_PROW_E2E_ALPHA_GATES_LATEST. +configvar CSI_PROW_E2E_ALPHA_GATES_LATEST 'VolumeSnapshotDataSource=true,ExpandCSIVolumes=true' "alpha feature gates for latest Kubernetes" configvar CSI_PROW_E2E_ALPHA_GATES "$(get_versioned_variable CSI_PROW_E2E_ALPHA_GATES "${csi_prow_kubernetes_version_suffix}")" "alpha E2E feature gates" # Some tests are known to be unusable in a KinD cluster. For example, From ff9bce4a71feb31e69d3e5a8ed9874da542066f1 Mon Sep 17 00:00:00 2001 From: Pengzhi Sun Date: Thu, 11 Apr 2019 17:42:44 +0800 Subject: [PATCH 037/107] Replace 'return' to 'exit' to fix shellcheck error --- build.make | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/build.make b/build.make index e5769c5..e3d4795 100644 --- a/build.make +++ b/build.make @@ -143,11 +143,11 @@ test-shellcheck: @ ret=0; \ if ! command -v docker; then \ echo "skipped, no Docker"; \ - return 0; \ + exit 0; \ fi; \ for dir in $(abspath $(TEST_SHELLCHECK_DIRS)); do \ echo; \ echo "$$dir:"; \ ./release-tools/verify-shellcheck.sh "$$dir" || ret=1; \ done; \ - return $$ret + exit $$ret From 0b10f6a4b026a52bdda5d8d63b80a45b79b84ff3 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Thu, 11 Apr 2019 13:37:43 +0200 Subject: [PATCH 038/107] prow.sh: update csi-driver-host-path This pulls in https://github.com/kubernetes-csi/csi-driver-host-path/pull/37, which turned out to be necessary for pre-submit testing of the livenessprobe. --- prow.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prow.sh b/prow.sh index 57415d4..ab3a1a5 100755 --- a/prow.sh +++ b/prow.sh @@ -151,7 +151,7 @@ configvar CSI_PROW_WORK "$(mkdir -p "$GOPATH/pkg" && mktemp -d "$GOPATH/pkg/csip # # When no deploy script is found (nothing in `deploy` directory, # CSI_PROW_HOSTPATH_REPO=none), nothing gets deployed. -configvar CSI_PROW_HOSTPATH_VERSION 486074dc3beef59955faf7bb5210418d9844e0a7 "hostpath driver" # pre-1.1.0 +configvar CSI_PROW_HOSTPATH_VERSION fc52d13ba07922c80555a24616a5b16480350c3f "hostpath driver" # pre-1.1.0 configvar CSI_PROW_HOSTPATH_REPO https://github.com/kubernetes-csi/csi-driver-host-path "hostpath repo" configvar CSI_PROW_DEPLOYMENT "" "deployment" From 0fafc663dc8c0d43cbc411b69ad68a74ecacbc8b Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Thu, 11 Apr 2019 22:32:01 +0200 Subject: [PATCH 039/107] prow.sh: skip sanity testing if component doesn't support it Whether a component supports sanity testing depends on the component. For example, csi-driver-host-path enables it because it makes sense there (and only there). Letting the prow.sh script decide whether it actually runs simplifies the job definitions in test-infra. --- prow.sh | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/prow.sh b/prow.sh index ab3a1a5..204cc29 100755 --- a/prow.sh +++ b/prow.sh @@ -198,10 +198,13 @@ configvar CSI_PROW_SANITY_CONTAINER "hostpath" "Kubernetes container with CSI dr # - serial, only alpha features # - sanity # -# Sanity testing with csi-sanity only covers the CSI driver itself and thus -# is off by default. A CSI driver can change that default in its .prow.sh -# by setting CSI_PROW_TESTS_SANITY. -configvar CSI_PROW_TESTS "unit parallel serial parallel-alpha serial-alpha ${CSI_PROW_TESTS_SANITY}" "tests to run" +# Unknown or unsupported entries are ignored. +# +# Sanity testing with csi-sanity only covers the CSI driver itself and +# thus only makes sense in repos which provide their own CSI +# driver. Repos can enable sanity testing by setting +# CSI_PROW_TESTS_SANITY=sanity. +configvar CSI_PROW_TESTS "unit parallel serial parallel-alpha serial-alpha sanity" "tests to run" tests_enabled () { local t1 t2 # We want word-splitting here, so ignore: Quote to prevent word splitting, or split robustly with mapfile or read -a. @@ -216,11 +219,16 @@ tests_enabled () { done return 1 } +sanity_enabled () { + [ "${CSI_PROW_TESTS_SANITY}" = "sanity" ] && tests_enabled "sanity" +} tests_need_kind () { - tests_enabled "sanity" "parallel" "serial" "serial-alpha" "parallel-alpha" + tests_enabled "parallel" "serial" "serial-alpha" "parallel-alpha" || + sanity_enabled } tests_need_non_alpha_cluster () { - tests_enabled "sanity" "parallel" "serial" + tests_enabled "parallel" "serial" || + sanity_enabled } tests_need_alpha_cluster () { tests_enabled "parallel-alpha" "serial-alpha" @@ -916,7 +924,7 @@ main () { if install_hostpath "$images"; then collect_cluster_info - if tests_enabled "sanity"; then + if sanity_enabled; then if ! run_sanity; then ret=1 fi From e157b6b514ace191f46d4c879491cb7d11684659 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Mon, 15 Apr 2019 09:45:23 +0200 Subject: [PATCH 040/107] update to Go 1.12.4 Kubernetes also requires 1.12. We pick the latest stable release for CI builds. --- travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/travis.yml b/travis.yml index 1c05dfd..5b04252 100644 --- a/travis.yml +++ b/travis.yml @@ -4,7 +4,7 @@ services: - docker matrix: include: - - go: 1.11.1 + - go: 1.12.4 before_script: - mkdir -p bin - wget https://github.com/golang/dep/releases/download/v0.5.1/dep-linux-amd64 -O bin/dep From 066143d146253748b9391445b25804e85b5cc208 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Fri, 10 May 2019 17:37:49 +0200 Subject: [PATCH 041/107] build.make: allow repos to use 'go mod' for vendoring How a repo does vendoring is detected based on the presence of Gopkg.toml. The vendor check with `dep` was all done locally, but the corresponding check for `go mod` requires network access. The check therefore gets skipped when running in the Prow CI in situations where we are sure that it isn't needed (for example, in a periodic job). --- build.make | 33 +++++++++++++++++++++++++++++---- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/build.make b/build.make index e3d4795..142c857 100644 --- a/build.make +++ b/build.make @@ -118,14 +118,39 @@ test-fmt: fi # This test only runs when dep >= 0.5 is installed, which is the case for the CI setup. +# When using 'go mod', we allow the test to be skipped in the Prow CI under some special +# circumstances, because it depends on accessing all remote repos and thus +# running it all the time would defeat the purpose of vendoring: +# - not handling a PR or +# - the fabricated merge commit leaves go.mod, go.sum and vendor dir unchanged +# - release-tools also didn't change (changing rules or Go version might lead to +# a different result and thus must be tested) .PHONY: test-vendor test: test-vendor test-vendor: @ echo; echo "### $@:" - @ case "$$(dep version 2>/dev/null | grep 'version *:')" in \ - *v0.[56789]*) dep check && echo "vendor up-to-date" || false;; \ - *) echo "skipping check, dep >= 0.5 required";; \ - esac + @ if [ -f Gopkg.toml ]; then \ + echo "Repo uses 'dep' for vendoring."; \ + case "$$(dep version 2>/dev/null | grep 'version *:')" in \ + *v0.[56789]*) dep check && echo "vendor up-to-date" || false;; \ + *) echo "skipping check, dep >= 0.5 required";; \ + esac; \ + else \ + echo "Repo uses 'go mod' for vendoring."; \ + if [ "$${JOB_NAME}" ] && \ + ( [ "$${JOB_TYPE}" != "presubmit" ] || \ + [ $$(git diff "${PULL_BASE_SHA}..HEAD" -- go.mod go.sum vendor release-tools | wc -l) -eq 0 ] ); then \ + echo "Skipping vendor check because the Prow pre-submit job does not change vendoring."; \ + elif ! GO111MODULE=on go mod vendor; then \ + echo "ERROR: vendor check failed."; \ + false; \ + elif [ $$(git status --porcelain -- vendor | wc -l) -gt 0 ]; then \ + echo "ERROR: vendor directory *not* up-to-date, it did get modified by 'GO111MODULE=on go mod vendor':"; \ + git status -- vendor; \ + git diff -- vendor; \ + false; \ + fi; \ + fi; .PHONY: test-subtree test: test-subtree From b2f4e051d4ffeadffc0980db745a33bacf84f36c Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Mon, 8 Jul 2019 20:36:55 +0200 Subject: [PATCH 042/107] prow.sh: flexible test driver config By moving the code into a separate function, other CSI drivers have a chance to overwrite it. For the hostpath driver itself we need the ability to set the driver name depending on which revision is getting installed. --- prow.sh | 44 ++++++++++++++++++++++++++------------------ 1 file changed, 26 insertions(+), 18 deletions(-) diff --git a/prow.sh b/prow.sh index 204cc29..dd33554 100755 --- a/prow.sh +++ b/prow.sh @@ -154,6 +154,7 @@ configvar CSI_PROW_WORK "$(mkdir -p "$GOPATH/pkg" && mktemp -d "$GOPATH/pkg/csip configvar CSI_PROW_HOSTPATH_VERSION fc52d13ba07922c80555a24616a5b16480350c3f "hostpath driver" # pre-1.1.0 configvar CSI_PROW_HOSTPATH_REPO https://github.com/kubernetes-csi/csi-driver-host-path "hostpath repo" configvar CSI_PROW_DEPLOYMENT "" "deployment" +configvar CSI_PROW_HOSTPATH_DRIVER_NAME "csi-hostpath" "the driver (aka provisioner) name of the chosen hostpath driver" # If CSI_PROW_HOSTPATH_CANARY is set (typically to "canary", but also # "1.0-canary"), then all image versions are replaced with that @@ -673,6 +674,29 @@ hostpath_supports_block () { echo "${result:-true}" } +# The default implementation of this function generates a external +# driver test configuration for the hostpath driver. +# +# The content depends on both what the E2E suite expects and what the +# installed hostpath driver supports. Generating it here seems prone +# to breakage, but it is uncertain where a better place might be. +generate_test_driver () { + cat <"${CSI_PROW_WORK}/hostpath-test-driver.yaml" <"${CSI_PROW_WORK}/test-driver.yaml" || die "generating test-driver.yaml failed" # Rename, merge and filter JUnit files. Necessary in case that we run the E2E suite again # and to avoid the large number of "skipped" tests that we get from using @@ -727,7 +735,7 @@ EOF trap move_junit EXIT cd "${GOPATH}/src/${CSI_PROW_E2E_IMPORT_PATH}" && - run_with_loggers ginkgo -v "$@" "${CSI_PROW_WORK}/e2e.test" -- -report-dir "${ARTIFACTS}" -storage.testdriver="${CSI_PROW_WORK}/hostpath-test-driver.yaml" + run_with_loggers ginkgo -v "$@" "${CSI_PROW_WORK}/e2e.test" -- -report-dir "${ARTIFACTS}" -storage.testdriver="${CSI_PROW_WORK}/test-driver.yaml" ) # Run csi-sanity against installed CSI driver. From a6f21d4054a1b70f9efd957a6eebe8f85892b666 Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Thu, 11 Jul 2019 14:33:47 -0700 Subject: [PATCH 043/107] Add variables for 1.15 --- prow.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/prow.sh b/prow.sh index dd33554..fc562b0 100755 --- a/prow.sh +++ b/prow.sh @@ -170,6 +170,7 @@ configvar CSI_PROW_HOSTPATH_CANARY "" "hostpath image" # CSI_PROW_E2E_REPO=none disables E2E testing. configvar CSI_PROW_E2E_VERSION_1_13 v1.14.0 "E2E version for Kubernetes 1.13.x" # we can't use the one from 1.13.x because it didn't have --storage.testdriver configvar CSI_PROW_E2E_VERSION_1_14 v1.14.0 "E2E version for Kubernetes 1.14.x" +configvar CSI_PROW_E2E_VERSION_1_15 v1.15.0 "E2E version for Kubernetes 1.15.x" # TODO: add new CSI_PROW_E2E_VERSION entry for future Kubernetes releases configvar CSI_PROW_E2E_VERSION_LATEST master "E2E version for Kubernetes master" # testing against Kubernetes master is already tracking a moving target, so we might as well use a moving E2E version configvar CSI_PROW_E2E_REPO_LATEST https://github.com/kubernetes/kubernetes "E2E repo for Kubernetes >= 1.13.x" # currently the same for all versions @@ -277,6 +278,7 @@ configvar CSI_PROW_E2E_ALPHA "$(get_versioned_variable CSI_PROW_E2E_ALPHA "${csi # it anymore for older releases. configvar CSI_PROW_E2E_ALPHA_GATES_1_13 'VolumeSnapshotDataSource=true,BlockVolume=true,CSIBlockVolume=true' "alpha feature gates for Kubernetes 1.13" configvar CSI_PROW_E2E_ALPHA_GATES_1_14 'VolumeSnapshotDataSource=true,ExpandCSIVolumes=true' "alpha feature gates for Kubernetes 1.14" +configvar CSI_PROW_E2E_ALPHA_GATES_1_15 'VolumeSnapshotDataSource=true,ExpandCSIVolumes=true' "alpha feature gates for Kubernetes 1.15" # TODO: add new CSI_PROW_ALPHA_GATES_xxx entry for future Kubernetes releases and # add new gates to CSI_PROW_E2E_ALPHA_GATES_LATEST. configvar CSI_PROW_E2E_ALPHA_GATES_LATEST 'VolumeSnapshotDataSource=true,ExpandCSIVolumes=true' "alpha feature gates for latest Kubernetes" From ecc79187c50d2c9aa2f27851b2dc5a59190032e2 Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Thu, 11 Jul 2019 14:42:59 -0700 Subject: [PATCH 044/107] Update kind to v0.4.0. This requires overriding Kubernetes versions with specific patch versions that kind 0.4.0 supports. Also, feature gate setting is only supported on 1.15+ due to kind.sigs.k8s.io/v1alpha3 and kubeadm.k8s.io/v1beta2 dependencies. --- prow.sh | 105 +++++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 74 insertions(+), 31 deletions(-) diff --git a/prow.sh b/prow.sh index fc562b0..be49ff5 100755 --- a/prow.sh +++ b/prow.sh @@ -52,6 +52,26 @@ configvar () { eval echo "\$3:" "$1=\${$1}" } +# Takes the minor version of $CSI_PROW_KUBERNETES_VERSION and overrides it to +# $1 if they are equal minor versions. Ignores versions that begin with +# "release-". +override_k8s_version () { + local current_minor_version + local override_minor_version + + # Ignore: See if you can use ${variable//search/replace} instead. + # shellcheck disable=SC2001 + current_minor_version="$(echo "${CSI_PROW_KUBERNETES_VERSION}" | sed -e 's/\([0-9]*\)\.\([0-9]*\).*/\1\.\2/')" + + # Ignore: See if you can use ${variable//search/replace} instead. + # shellcheck disable=SC2001 + override_minor_version="$(echo "${1}" | sed -e 's/\([0-9]*\)\.\([0-9]*\).*/\1\.\2/')" + if [ "${current_minor_version}" == "${override_minor_version}" ]; then + CSI_PROW_KUBERNETES_VERSION="$1" + echo "Overriding CSI_PROW_KUBERNETES_VERSION with $1: $CSI_PROW_KUBERNETES_VERSION" + fi +} + # Prints the value of a variable + version suffix, falling back to variable + "LATEST". get_versioned_variable () { local var="$1" @@ -81,7 +101,7 @@ configvar CSI_PROW_GO_VERSION_GINKGO "${CSI_PROW_GO_VERSION_BUILD}" "Go version # kind version to use. If the pre-installed version is different, # the desired version is downloaded from https://github.com/kubernetes-sigs/kind/releases/download/ # (if available), otherwise it is built from source. -configvar CSI_PROW_KIND_VERSION 0.2.1 "kind" +configvar CSI_PROW_KIND_VERSION v0.4.0 "kind" # ginkgo test runner version to use. If the pre-installed version is # different, the desired version is built from source. @@ -108,6 +128,18 @@ configvar CSI_PROW_BUILD_JOB true "building code in repo enabled" # deprecating or changing the implementation of an alpha feature. configvar CSI_PROW_KUBERNETES_VERSION 1.13.3 "Kubernetes" +# This is a hack to workaround the issue that each version +# of kind currently only supports specific patch versions of +# Kubernetes. We need to override CSI_PROW_KUBERNETES_VERSION +# passed in by our CI/pull jobs to the versions that +# kind v0.4.0 supports. +# +# If the version is prefixed with "release-", then nothing +# is overridden. +override_k8s_version "1.13.7" +override_k8s_version "1.14.3" +override_k8s_version "1.15.0" + # CSI_PROW_KUBERNETES_VERSION reduced to first two version numbers and # with underscore (1_13 instead of 1.13.3) and in uppercase (LATEST # instead of latest). @@ -466,40 +498,51 @@ start_cluster () { image="kindest/node:v${CSI_PROW_KUBERNETES_VERSION}" fi cat >"${CSI_PROW_WORK}/kind-config.yaml" <>"${CSI_PROW_WORK}/kind-config.yaml" < Date: Thu, 11 Jul 2019 14:57:27 -0700 Subject: [PATCH 045/107] Update hostpath version for sidecar testing to v1.2.0-rc2 --- prow.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prow.sh b/prow.sh index be49ff5..d998ebb 100755 --- a/prow.sh +++ b/prow.sh @@ -183,7 +183,7 @@ configvar CSI_PROW_WORK "$(mkdir -p "$GOPATH/pkg" && mktemp -d "$GOPATH/pkg/csip # # When no deploy script is found (nothing in `deploy` directory, # CSI_PROW_HOSTPATH_REPO=none), nothing gets deployed. -configvar CSI_PROW_HOSTPATH_VERSION fc52d13ba07922c80555a24616a5b16480350c3f "hostpath driver" # pre-1.1.0 +configvar CSI_PROW_HOSTPATH_VERSION "v1.2.0-rc2" "hostpath driver" configvar CSI_PROW_HOSTPATH_REPO https://github.com/kubernetes-csi/csi-driver-host-path "hostpath repo" configvar CSI_PROW_DEPLOYMENT "" "deployment" configvar CSI_PROW_HOSTPATH_DRIVER_NAME "csi-hostpath" "the driver (aka provisioner) name of the chosen hostpath driver" From 4e31f078a2fb7a891a7cbcdf2299b94891d030ce Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Thu, 11 Jul 2019 17:13:06 -0700 Subject: [PATCH 046/107] Change default hostpath driver name to hostpath.csi.k8s.io --- prow.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prow.sh b/prow.sh index d998ebb..f1f0a1b 100755 --- a/prow.sh +++ b/prow.sh @@ -186,7 +186,7 @@ configvar CSI_PROW_WORK "$(mkdir -p "$GOPATH/pkg" && mktemp -d "$GOPATH/pkg/csip configvar CSI_PROW_HOSTPATH_VERSION "v1.2.0-rc2" "hostpath driver" configvar CSI_PROW_HOSTPATH_REPO https://github.com/kubernetes-csi/csi-driver-host-path "hostpath repo" configvar CSI_PROW_DEPLOYMENT "" "deployment" -configvar CSI_PROW_HOSTPATH_DRIVER_NAME "csi-hostpath" "the driver (aka provisioner) name of the chosen hostpath driver" +configvar CSI_PROW_HOSTPATH_DRIVER_NAME "hostpath.csi.k8s.io" "the hostpath driver name" # If CSI_PROW_HOSTPATH_CANARY is set (typically to "canary", but also # "1.0-canary"), then all image versions are replaced with that From 0c0dc300cafca09a27dd7206547a8191baae477e Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Mon, 15 Jul 2019 12:42:20 -0700 Subject: [PATCH 047/107] prow.sh: tag master images with a large version number --- prow.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prow.sh b/prow.sh index f1f0a1b..dcfcec6 100755 --- a/prow.sh +++ b/prow.sh @@ -485,7 +485,7 @@ start_cluster () { tag="$(echo "${CSI_PROW_KUBERNETES_VERSION}" | sed -e 's/release-\(.*\)/v\1.0-release./')";; *) # We have to make something up. v1.0.0 did not work for some reasons. - tag="v1.14.0-";; + tag="v999.999.999-";; esac tag="$tag$(cd "$GOPATH/src/k8s.io/kubernetes" && git rev-list --abbrev-commit HEAD).csiprow" (cd "$GOPATH/src/k8s.io/kubernetes" && run git tag -f "$tag") || die "git tag failed" From f46191d9a8924babaa9c74c7620c5f4caa5cae24 Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Mon, 5 Aug 2019 09:45:55 -0700 Subject: [PATCH 048/107] Kubernetes master changed the way that releases are tagged, which needed changes to kind. There are 3 changes made to prow.sh: 1. Use a master commit of kind that includes the fix for Kubernetes master. 2. Use git clone instead of git checkout (shallow) to source Kubernetes. This lets kind correctly figure out the Kubernetes release tag. 3. Build kind with make install. The kind fix was not working correctly when built with go build. --- prow.sh | 52 +++++++++++++++++++++++++++++----------------------- 1 file changed, 29 insertions(+), 23 deletions(-) diff --git a/prow.sh b/prow.sh index dcfcec6..fd32f79 100755 --- a/prow.sh +++ b/prow.sh @@ -101,7 +101,7 @@ configvar CSI_PROW_GO_VERSION_GINKGO "${CSI_PROW_GO_VERSION_BUILD}" "Go version # kind version to use. If the pre-installed version is different, # the desired version is downloaded from https://github.com/kubernetes-sigs/kind/releases/download/ # (if available), otherwise it is built from source. -configvar CSI_PROW_KIND_VERSION v0.4.0 "kind" +configvar CSI_PROW_KIND_VERSION 2555d8e09d5a77ee718414cec9f6083dfa028dc5 "kind" # ginkgo test runner version to use. If the pre-installed version is # different, the desired version is built from source. @@ -322,7 +322,7 @@ configvar CSI_PROW_E2E_ALPHA_GATES "$(get_versioned_variable CSI_PROW_E2E_ALPHA_ # whether they can run with the current cluster provider, but until # they are, we filter them out by name. Like the other test selection # variables, this is again a space separated list of regular expressions. -configvar CSI_PROW_E2E_SKIP 'while.kubelet.is.down.*Disruptive' "tests that need to be skipped" +configvar CSI_PROW_E2E_SKIP 'Disruptive' "tests that need to be skipped" # This is the directory for additional result files. Usually set by Prow, but # if not (for example, when invoking manually) it defaults to the work directory. @@ -379,8 +379,8 @@ install_kind () { if run curl --fail --location -o "${CSI_PROW_WORK}/bin/kind" "https://github.com/kubernetes-sigs/kind/releases/download/${CSI_PROW_KIND_VERSION}/kind-linux-amd64"; then chmod u+x "${CSI_PROW_WORK}/bin/kind" else - git_checkout https://github.com/kubernetes-sigs/kind "$GOPATH/src/sigs.k8s.io/kind" "${CSI_PROW_KIND_VERSION}" --depth=1 && - run_with_go "${CSI_PROW_GO_VERSION_KIND}" go build -o "${CSI_PROW_WORK}/bin/kind" sigs.k8s.io/kind + git_checkout https://github.com/kubernetes-sigs/kind "${GOPATH}/src/sigs.k8s.io/kind" "${CSI_PROW_KIND_VERSION}" --depth=1 && + (cd "${GOPATH}/src/sigs.k8s.io/kind" && make install INSTALL_DIR="${CSI_PROW_WORK}/bin") fi } @@ -427,6 +427,27 @@ git_checkout () { (cd "$path" && run git clean -fdx) || die "failed to clean $path" } +# This clones a repo ("https://github.com/kubernetes/kubernetes") +# in a certain location ("$GOPATH/src/k8s.io/kubernetes") at +# a the head of a specific branch (i.e., release-1.13, master). +# The directory cannot exist. +git_clone_branch () { + local repo path branch parent + repo="$1" + shift + path="$1" + shift + branch="$1" + shift + + parent="$(dirname "$path")" + mkdir -p "$parent" + (cd "$parent" && run git clone --single-branch --branch "$branch" "$repo" "$path") || die "cloning $repo" failed + # This is useful for local testing or when switching between different revisions in the same + # repo. + (cd "$path" && run git clean -fdx) || die "failed to clean $path" +} + list_gates () ( set -f; IFS=',' # Ignore: Double quote to prevent globbing and word splitting. @@ -472,25 +493,10 @@ start_cluster () { if [ "$version" = "latest" ]; then version=master fi - git_checkout https://github.com/kubernetes/kubernetes "$GOPATH/src/k8s.io/kubernetes" "$version" --depth=1 || die "checking out Kubernetes $version failed" - - # "kind build" and/or the Kubernetes build rules need at least one tag, which we don't have - # when doing a shallow fetch. Therefore we fake one: - # release-1.12 -> v1.12.0-release..csiprow - # latest or -> v1.14.0-.csiprow - case "${CSI_PROW_KUBERNETES_VERSION}" in - release-*) - # Ignore: See if you can use ${variable//search/replace} instead. - # shellcheck disable=SC2001 - tag="$(echo "${CSI_PROW_KUBERNETES_VERSION}" | sed -e 's/release-\(.*\)/v\1.0-release./')";; - *) - # We have to make something up. v1.0.0 did not work for some reasons. - tag="v999.999.999-";; - esac - tag="$tag$(cd "$GOPATH/src/k8s.io/kubernetes" && git rev-list --abbrev-commit HEAD).csiprow" - (cd "$GOPATH/src/k8s.io/kubernetes" && run git tag -f "$tag") || die "git tag failed" - go_version="$(go_version_for_kubernetes "$GOPATH/src/k8s.io/kubernetes" "$version")" || die "cannot proceed without knowing Go version for Kubernetes" - run_with_go "$go_version" kind build node-image --type bazel --image csiprow/node:latest --kube-root "$GOPATH/src/k8s.io/kubernetes" || die "'kind build node-image' failed" + git_clone_branch https://github.com/kubernetes/kubernetes "${CSI_PROW_WORK}/src/kubernetes" "$version" || die "checking out Kubernetes $version failed" + + go_version="$(go_version_for_kubernetes "${CSI_PROW_WORK}/src/kubernetes" "$version")" || die "cannot proceed without knowing Go version for Kubernetes" + run_with_go "$go_version" kind build node-image --type bazel --image csiprow/node:latest --kube-root "${CSI_PROW_WORK}/src/kubernetes" || die "'kind build node-image' failed" csi_prow_kind_have_kubernetes=true fi image="csiprow/node:latest" From dc0a5d8380459b2ddb8ac11307dd91749fd43b99 Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Wed, 21 Aug 2019 15:19:39 -0700 Subject: [PATCH 049/107] Update kind to v0.5.0 --- prow.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/prow.sh b/prow.sh index fd32f79..c334a5d 100755 --- a/prow.sh +++ b/prow.sh @@ -101,7 +101,7 @@ configvar CSI_PROW_GO_VERSION_GINKGO "${CSI_PROW_GO_VERSION_BUILD}" "Go version # kind version to use. If the pre-installed version is different, # the desired version is downloaded from https://github.com/kubernetes-sigs/kind/releases/download/ # (if available), otherwise it is built from source. -configvar CSI_PROW_KIND_VERSION 2555d8e09d5a77ee718414cec9f6083dfa028dc5 "kind" +configvar CSI_PROW_KIND_VERSION "v0.5.0" "kind" # ginkgo test runner version to use. If the pre-installed version is # different, the desired version is built from source. @@ -126,19 +126,19 @@ configvar CSI_PROW_BUILD_JOB true "building code in repo enabled" # use the same settings as for "latest" Kubernetes. This works # as long as there are no breaking changes in Kubernetes, like # deprecating or changing the implementation of an alpha feature. -configvar CSI_PROW_KUBERNETES_VERSION 1.13.3 "Kubernetes" +configvar CSI_PROW_KUBERNETES_VERSION 1.15.3 "Kubernetes" # This is a hack to workaround the issue that each version # of kind currently only supports specific patch versions of # Kubernetes. We need to override CSI_PROW_KUBERNETES_VERSION # passed in by our CI/pull jobs to the versions that -# kind v0.4.0 supports. +# kind v0.5.0 supports. # # If the version is prefixed with "release-", then nothing # is overridden. -override_k8s_version "1.13.7" -override_k8s_version "1.14.3" -override_k8s_version "1.15.0" +override_k8s_version "1.13.10" +override_k8s_version "1.14.6" +override_k8s_version "1.15.3" # CSI_PROW_KUBERNETES_VERSION reduced to first two version numbers and # with underscore (1_13 instead of 1.13.3) and in uppercase (LATEST From 9fba09b41405b93a9bad8cd0d39544e21cdc99e3 Mon Sep 17 00:00:00 2001 From: Deep Debroy Date: Wed, 28 Aug 2019 06:24:26 -0700 Subject: [PATCH 050/107] Add rule for building Windows binaries Signed-off-by: Deep Debroy --- build.make | 1 + 1 file changed, 1 insertion(+) diff --git a/build.make b/build.make index 142c857..f8b737f 100644 --- a/build.make +++ b/build.make @@ -63,6 +63,7 @@ endif build-%: mkdir -p bin CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$* ./cmd/$* + CGO_ENABLED=0 GOOS=windows go build -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$* ./cmd/$* container-%: build-% docker build -t $*:latest -f $(shell if [ -e ./cmd/$*/Dockerfile ]; then echo ./cmd/$*/Dockerfile; else echo Dockerfile; fi) --label revision=$(REV) . From 35ceaedcbddf39c13651ae07ca8fd73083d4f3e4 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Thu, 29 Aug 2019 15:00:21 +0200 Subject: [PATCH 051/107] prow.sh: install dep if needed "make test-vendor" depends on dep if the current project uses dep. Without it, the vendor directory checking was skipped under Prow. --- prow.sh | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/prow.sh b/prow.sh index c334a5d..33d93e4 100755 --- a/prow.sh +++ b/prow.sh @@ -223,6 +223,10 @@ configvar CSI_PROW_SANITY_SERVICE "hostpath-service" "Kubernetes TCP service nam configvar CSI_PROW_SANITY_POD "csi-hostpathplugin-0" "Kubernetes pod with CSI driver" configvar CSI_PROW_SANITY_CONTAINER "hostpath" "Kubernetes container with CSI driver" +# The version of dep to use for 'make test-vendor'. Ignored if the project doesn't +# use dep. Only binary releases of dep are supported (https://github.com/golang/dep/releases). +configvar CSI_PROW_DEP_VERSION v0.5.1 "golang dep version to be used for vendor checking" + # Each job can run one or more of the following tests, identified by # a single word: # - unit testing @@ -396,6 +400,15 @@ install_ginkgo () { mv "$GOPATH/bin/ginkgo" "${CSI_PROW_BIN}" } +# Ensure that we have the desired version of dep. +install_dep () { + if dep version 2>/dev/null | grep -q "version:.*${CSI_PROW_DEP_VERSION}$"; then + return + fi + run curl --fail --location -o "${CSI_PROW_WORK}/bin/dep" "https://github.com/golang/dep/releases/download/v0.5.4/dep-linux-amd64" && + chmod u+x "${CSI_PROW_WORK}/bin/dep" +} + # This checks out a repo ("https://github.com/kubernetes/kubernetes") # in a certain location ("$GOPATH/src/k8s.io/kubernetes") at # a certain revision (a hex commit hash, v1.13.1, master). It's okay @@ -936,6 +949,10 @@ main () { # changes in "release-tools" in a PR (that fails the "is release-tools unmodified" # test). if tests_enabled "unit"; then + if [ -f Gopkg.toml ] && ! install_dep; then + warn "installing 'dep' failed, cannot test vendoring" + ret=1 + fi if ! run_with_go "${CSI_PROW_GO_VERSION_BUILD}" make -k test 2>&1 | make_test_to_junit; then warn "'make test' failed, proceeding anyway" ret=1 From a8ea8bcc5c6b440cb85cb5341f751b63e063fe62 Mon Sep 17 00:00:00 2001 From: Mucahit Kurt Date: Sat, 7 Sep 2019 02:31:15 +0300 Subject: [PATCH 052/107] create 2-node kind cluster since topology support is added to hostpath driver Signed-off-by: Mucahit Kurt --- prow.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/prow.sh b/prow.sh index 33d93e4..5e7277a 100755 --- a/prow.sh +++ b/prow.sh @@ -521,6 +521,7 @@ kind: Cluster apiVersion: kind.sigs.k8s.io/v1alpha3 nodes: - role: control-plane +- role: worker EOF # kubeadm has API dependencies between apiVersion and Kubernetes version From ea2f1b5277eef96e0855d4d817ceae832079f59b Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Fri, 6 Sep 2019 19:15:35 -0700 Subject: [PATCH 053/107] build windows binaries with .exe suffix --- build.make | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.make b/build.make index f8b737f..36e19be 100644 --- a/build.make +++ b/build.make @@ -63,7 +63,7 @@ endif build-%: mkdir -p bin CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$* ./cmd/$* - CGO_ENABLED=0 GOOS=windows go build -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$* ./cmd/$* + CGO_ENABLED=0 GOOS=windows go build -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$*.exe ./cmd/$* container-%: build-% docker build -t $*:latest -f $(shell if [ -e ./cmd/$*/Dockerfile ]; then echo ./cmd/$*/Dockerfile; else echo Dockerfile; fi) --label revision=$(REV) . From 6208f6ab47bc2d2973db3c276fd270fe13ac60b5 Mon Sep 17 00:00:00 2001 From: Hemant Kumar Date: Mon, 16 Sep 2019 17:59:04 -0400 Subject: [PATCH 054/107] Enable hostpath expansion --- prow.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/prow.sh b/prow.sh index 5e7277a..d93b6fa 100755 --- a/prow.sh +++ b/prow.sh @@ -183,7 +183,7 @@ configvar CSI_PROW_WORK "$(mkdir -p "$GOPATH/pkg" && mktemp -d "$GOPATH/pkg/csip # # When no deploy script is found (nothing in `deploy` directory, # CSI_PROW_HOSTPATH_REPO=none), nothing gets deployed. -configvar CSI_PROW_HOSTPATH_VERSION "v1.2.0-rc2" "hostpath driver" +configvar CSI_PROW_HOSTPATH_VERSION "v1.2.0-rc8" "hostpath driver" configvar CSI_PROW_HOSTPATH_REPO https://github.com/kubernetes-csi/csi-driver-host-path "hostpath repo" configvar CSI_PROW_DEPLOYMENT "" "deployment" configvar CSI_PROW_HOSTPATH_DRIVER_NAME "hostpath.csi.k8s.io" "the hostpath driver name" @@ -759,6 +759,8 @@ DriverInfo: persistence: true dataSource: true multipods: true + nodeExpansion: true + controllerExpansion: true EOF } From 194289aa8eece02c19f537b26a361932cc0ff73d Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Fri, 4 Oct 2019 14:08:31 +0200 Subject: [PATCH 055/107] update Go mod support It turned out that changes like https://github.com/kubernetes-csi/csi-lib-utils/pull/33 should better have been committed after `go mod tidy` because that adds some indirect dependencies in that example. The revised `test-vendor` checks for that and (just in case that this ever becomes desired) allows projects to not have a vendor directory when using `go mod`. How to use `go mod` properly gets documented in the README.md, because there are such pitfalls. --- README.md | 34 +++++++++++++++++++++++++++++++++ build.make | 55 ++++++++++++++++++++++++++++++++++++++++++++---------- 2 files changed, 79 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index bc061ae..d5fdb3a 100644 --- a/README.md +++ b/README.md @@ -106,3 +106,37 @@ Kubernetes releases: CSI_PROW_KUBERNETES_VERSION=1.13.3 ./.prow.sh CSI_PROW_KUBERNETES_VERSION=latest ./.prow.sh + +Dependencies and vendoring +-------------------------- + +Most projects will (eventually) use `go mod` to manage +dependencies. `dep` is also still supported by `csi-release-tools`, +but not documented here because it's not recommended anymore. + +The usual instructions for using [go +modules](https://github.com/golang/go/wiki/Modules) apply. Here's a cheat sheet +for some of the relevant commands: +- list available updates: `GO111MODULE=on go list -u -m all` +- update or add a single dependency: `GO111MODULE=on go get ` +- update all dependencies to their next minor or patch release: + `GO111MODULE=on go get ./...` (add `-u=patch` to limit to patch + releases) +- lock onto a specific version: `GO111MODULE=on go get @` +- clean up `go.mod`: `GO111MODULE=on go mod tidy` +- update vendor directory: `GO111MODULE=on go mod vendor` + +`GO111MODULE=on` can be left out when using Go >= 1.13 or when the +source is checked out outside of `$GOPATH`. + +`go mod tidy` must be used to ensure that the listed dependencies are +really still needed. Changing import statements or a tentative `go +get` can result in stale dependencies. + +The `test-vendor` verifies that it was used when run locally or in a +pre-merge CI job. If a `vendor` directory is present, it will also +verify that it's content is up-to-date. + +The `vendor` directory is optional. It is still present in projects +because it avoids downloading sources during CI builds. If this is no +longer deemed necessary, then a project can also remove the directory. diff --git a/build.make b/build.make index 142c857..c8384c1 100644 --- a/build.make +++ b/build.make @@ -125,6 +125,26 @@ test-fmt: # - the fabricated merge commit leaves go.mod, go.sum and vendor dir unchanged # - release-tools also didn't change (changing rules or Go version might lead to # a different result and thus must be tested) +# - import statements not changed (because if they change, go.mod might have to be updated) +# +# "git diff" is intelligent enough to annotate changes inside the "import" block in +# the start of the diff hunk: +# +# diff --git a/rpc/common.go b/rpc/common.go +# index bb4a5c4..5fa4271 100644 +# --- a/rpc/common.go +# +++ b/rpc/common.go +# @@ -21,7 +21,6 @@ import ( +# "fmt" +# "time" +# +# - "google.golang.org/grpc" +# "google.golang.org/grpc/codes" +# "google.golang.org/grpc/status" +# +# We rely on that to find such changes. +# +# Vendoring is optional when using go.mod. .PHONY: test-vendor test: test-vendor test-vendor: @@ -135,22 +155,37 @@ test-vendor: *v0.[56789]*) dep check && echo "vendor up-to-date" || false;; \ *) echo "skipping check, dep >= 0.5 required";; \ esac; \ - else \ - echo "Repo uses 'go mod' for vendoring."; \ + elif [ -f go.mod ]; then \ + echo "Repo uses 'go mod'."; \ if [ "$${JOB_NAME}" ] && \ ( [ "$${JOB_TYPE}" != "presubmit" ] || \ - [ $$(git diff "${PULL_BASE_SHA}..HEAD" -- go.mod go.sum vendor release-tools | wc -l) -eq 0 ] ); then \ - echo "Skipping vendor check because the Prow pre-submit job does not change vendoring."; \ - elif ! GO111MODULE=on go mod vendor; then \ + [ $$( (git diff "${PULL_BASE_SHA}..HEAD" -- go.mod go.sum vendor release-tools; \ + git diff "${PULL_BASE_SHA}..HEAD" | grep -e '^@@.*@@ import (' -e '^[+-]import') | \ + wc -l) -eq 0 ] ); then \ + echo "Skipping vendor check because the Prow pre-submit job does not affect dependencies."; \ + elif ! GO111MODULE=on go mod tidy; then \ echo "ERROR: vendor check failed."; \ false; \ - elif [ $$(git status --porcelain -- vendor | wc -l) -gt 0 ]; then \ - echo "ERROR: vendor directory *not* up-to-date, it did get modified by 'GO111MODULE=on go mod vendor':"; \ - git status -- vendor; \ - git diff -- vendor; \ + elif [ $$(git status --porcelain -- go.mod go.sum | wc -l) -gt 0 ]; then \ + echo "ERROR: go module files *not* up-to-date, they did get modified by 'GO111MODULE=on go mod tidy':"; \ + git diff -- go.mod go.sum; \ false; \ + elif [ -d vendor ]; then \ + if ! GO111MODULE=on go mod vendor; then \ + echo "ERROR: vendor check failed."; \ + false; \ + elif [ $$(git status --porcelain -- vendor | wc -l) -gt 0 ]; then \ + echo "ERROR: vendor directory *not* up-to-date, it did get modified by 'GO111MODULE=on go mod vendor':"; \ + git status -- vendor; \ + git diff -- vendor; \ + false; \ + else \ + echo "Go dependencies and vendor directory up-to-date."; \ + fi; \ + else \ + echo "Go dependencies up-to-date."; \ fi; \ - fi; + fi .PHONY: test-subtree test: test-subtree From c1078a6585735bfcad4d273dee57860544d6fd2a Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Mon, 7 Oct 2019 12:21:09 +0200 Subject: [PATCH 056/107] go-get-kubernetes.sh: automate Kubernetes dependency handling This script handles the necessary "replace" statements and determines which packages need to be updated in lockstep. --- README.md | 20 +++++++++ go-get-kubernetes.sh | 104 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 124 insertions(+) create mode 100755 go-get-kubernetes.sh diff --git a/README.md b/README.md index d5fdb3a..cc40f16 100644 --- a/README.md +++ b/README.md @@ -140,3 +140,23 @@ verify that it's content is up-to-date. The `vendor` directory is optional. It is still present in projects because it avoids downloading sources during CI builds. If this is no longer deemed necessary, then a project can also remove the directory. + +When using packages that are part of the Kubernetes source code, the +commands above are not enough because the [lack of semantic +versioning](https://github.com/kubernetes/kubernetes/issues/72638) +prevents `go mod` from finding newer releases. Importing directly from +`kubernetes/kubernetes` also needs `replace` statements to override +the fake `v0.0.0` versions +(https://github.com/kubernetes/kubernetes/issues/79384). The +`go-get-kubernetes.sh` script can be used to update all packages in +lockstep to a different Kubernetes version. It takes a single version +number like "1.16.0". + +Conversion of a repository that uses `dep` to `go mod` can be done with: + + GO111MODULE=on go mod init + release-tools/go-get-kubernetes.sh + GO111MODULE=on go mod tidy + GO111MODULE=on go mod vendor + git rm -f Gopkg.toml Gopkg.lock + git add go.mod go.sum vendor diff --git a/go-get-kubernetes.sh b/go-get-kubernetes.sh new file mode 100755 index 0000000..8c4e302 --- /dev/null +++ b/go-get-kubernetes.sh @@ -0,0 +1,104 @@ +#!/usr/bin/env bash + +# Copyright 2019 The Kubernetes Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# This script can be used while converting a repo from "dep" to "go mod" +# by calling it after "go mod init" or to update the Kubernetes packages +# in a repo that has already been converted. Only packages that are +# part of kubernetes/kubernetes and thus part of a Kubernetes release +# are modified. Other k8.io packages (like k8s.io/klog, k8s.io/utils) +# need to be updated separately. + +set -o pipefail + +cmd=$0 + +function help () { + echo "$cmd - update all components from kubernetes/kubernetes to that version" +} + +if [ $# -ne 1 ]; then + help + exit 1 +fi +case "$1" in -h|--help|help) help; exit 0;; esac + +die () { + echo >&2 "$@" + exit 1 +} + +k8s="$1" + +# If the repo imports k8s.io/kubernetes (directly or indirectly), then +# "go mod" will try to find "v0.0.0" versions because +# k8s.io/kubernetes has those in it's go.mod file +# (https://github.com/kubernetes/kubernetes/blob/2bd9643cee5b3b3a5ecbd3af49d09018f0773c77/go.mod#L146-L157). +# (https://github.com/kubernetes/kubernetes/issues/79384). +# +# We need to replicate the replace statements to override those fake +# versions also in our go.mod file (idea and some code from +# https://github.com/kubernetes/kubernetes/issues/79384#issuecomment-521493597). +mods=$( (set -x; curl --silent --show-error --fail "https://raw.githubusercontent.com/kubernetes/kubernetes/v${k8s}/go.mod") | + sed -n 's|.*k8s.io/\(.*\) => ./staging/src/k8s.io/.*|k8s.io/\1|p' + ) || die "failed to determine Kubernetes staging modules" +for mod in $mods; do + # The presence of a potentially incomplete go.mod file affects this command, + # so move elsewhere. + modinfo=$(set -x; cd /; env GO111MODULE=on go mod download -json "$mod@kubernetes-${k8s}") || + die "failed to determine version of $mod: $modinfo" + v=$(echo "$modinfo" | sed -n 's|.*"Version": "\(.*\)".*|\1|p') + (set -x; env GO111MODULE=on go mod edit "-replace=$mod=$mod@$v") || die "'go mod edit' failed" +done + +packages= + +# Beware that we have to work with packages, not modules (i.e. no -m +# flag), because some modules trigger a "no Go code except tests" +# error. Getting their packages works. +if ! packages=$( (set -x; env GO111MODULE=on go list all) | grep ^k8s.io/ | sed -e 's; *;;'); then + cat >&2 <&2 <" go.mod; then + deps="$deps $(echo "$package" | sed -e "s;\$;@kubernetes-$k8s;" -e 's;^k8s.io/kubernetes\(/.*\)@kubernetes-;k8s.io/kubernetes\1@v;')" + fi +done + +# shellcheck disable=SC2086 +(set -x; env GO111MODULE=on go get $deps 2>&1) || die "go get failed" +echo "SUCCESS" From 2d6b3ce85f79a020c4929f8951840dcd1785434b Mon Sep 17 00:00:00 2001 From: Deep Debroy Date: Mon, 7 Oct 2019 16:38:04 -0700 Subject: [PATCH 057/107] Build Windows only for amd64 Signed-off-by: Deep Debroy --- build.make | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/build.make b/build.make index 36e19be..8d715e4 100644 --- a/build.make +++ b/build.make @@ -57,13 +57,17 @@ else TESTARGS = endif +ARCH := $(if $(GOARCH),$(GOARCH),$(shell go env GOARCH)) + # Specific packages can be excluded from each of the tests below by setting the *_FILTER_CMD variables # to something like "| grep -v 'github.com/kubernetes-csi/project/pkg/foobar'". See usage below. build-%: mkdir -p bin CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$* ./cmd/$* - CGO_ENABLED=0 GOOS=windows go build -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$*.exe ./cmd/$* + if [ "$$ARCH" = "amd64" ]; then \ + CGO_ENABLED=0 GOOS=windows go build -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$*.exe ./cmd/$* ; \ + fi container-%: build-% docker build -t $*:latest -f $(shell if [ -e ./cmd/$*/Dockerfile ]; then echo ./cmd/$*/Dockerfile; else echo Dockerfile; fi) --label revision=$(REV) . From f1697d2cac2e184d60b96d45720ed7a37a4ef9ed Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Tue, 8 Oct 2019 14:44:56 -0700 Subject: [PATCH 058/107] Do full git clones in travis. Shallow clones are causing test-subtree errors when the depth is exactly 50. --- travis.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/travis.yml b/travis.yml index 5b04252..a713bed 100644 --- a/travis.yml +++ b/travis.yml @@ -2,6 +2,8 @@ language: go sudo: required services: - docker +git: + depth: false matrix: include: - go: 1.12.4 From e0fde8c4f5684f5c259c8c7a1d116def587cabf0 Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Wed, 30 Oct 2019 18:08:41 -0700 Subject: [PATCH 059/107] Add new variables for 1.16 and remove 1.13 --- prow.sh | 38 ++++++++++---------------------------- 1 file changed, 10 insertions(+), 28 deletions(-) diff --git a/prow.sh b/prow.sh index d93b6fa..e3308e7 100755 --- a/prow.sh +++ b/prow.sh @@ -101,7 +101,8 @@ configvar CSI_PROW_GO_VERSION_GINKGO "${CSI_PROW_GO_VERSION_BUILD}" "Go version # kind version to use. If the pre-installed version is different, # the desired version is downloaded from https://github.com/kubernetes-sigs/kind/releases/download/ # (if available), otherwise it is built from source. -configvar CSI_PROW_KIND_VERSION "v0.5.0" "kind" +# TODO: https://github.com/kubernetes-csi/csi-release-tools/issues/39 +configvar CSI_PROW_KIND_VERSION "86bc23d84ac12dcb56a0528890736e2c347c2dc3" "kind" # ginkgo test runner version to use. If the pre-installed version is # different, the desired version is built from source. @@ -136,7 +137,6 @@ configvar CSI_PROW_KUBERNETES_VERSION 1.15.3 "Kubernetes" # # If the version is prefixed with "release-", then nothing # is overridden. -override_k8s_version "1.13.10" override_k8s_version "1.14.6" override_k8s_version "1.15.3" @@ -183,7 +183,7 @@ configvar CSI_PROW_WORK "$(mkdir -p "$GOPATH/pkg" && mktemp -d "$GOPATH/pkg/csip # # When no deploy script is found (nothing in `deploy` directory, # CSI_PROW_HOSTPATH_REPO=none), nothing gets deployed. -configvar CSI_PROW_HOSTPATH_VERSION "v1.2.0-rc8" "hostpath driver" +configvar CSI_PROW_HOSTPATH_VERSION "v1.2.0" "hostpath driver" configvar CSI_PROW_HOSTPATH_REPO https://github.com/kubernetes-csi/csi-driver-host-path "hostpath repo" configvar CSI_PROW_DEPLOYMENT "" "deployment" configvar CSI_PROW_HOSTPATH_DRIVER_NAME "hostpath.csi.k8s.io" "the hostpath driver name" @@ -200,9 +200,10 @@ configvar CSI_PROW_HOSTPATH_CANARY "" "hostpath image" # all generated files are present. # # CSI_PROW_E2E_REPO=none disables E2E testing. -configvar CSI_PROW_E2E_VERSION_1_13 v1.14.0 "E2E version for Kubernetes 1.13.x" # we can't use the one from 1.13.x because it didn't have --storage.testdriver +# TOOO: remove versioned variables and make e2e version match k8s version configvar CSI_PROW_E2E_VERSION_1_14 v1.14.0 "E2E version for Kubernetes 1.14.x" configvar CSI_PROW_E2E_VERSION_1_15 v1.15.0 "E2E version for Kubernetes 1.15.x" +configvar CSI_PROW_E2E_VERSION_1_16 v1.16.0 "E2E version for Kubernetes 1.16.x" # TODO: add new CSI_PROW_E2E_VERSION entry for future Kubernetes releases configvar CSI_PROW_E2E_VERSION_LATEST master "E2E version for Kubernetes master" # testing against Kubernetes master is already tracking a moving target, so we might as well use a moving E2E version configvar CSI_PROW_E2E_REPO_LATEST https://github.com/kubernetes/kubernetes "E2E repo for Kubernetes >= 1.13.x" # currently the same for all versions @@ -292,11 +293,6 @@ regex_join () { # alpha in previous Kubernetes releases. This was considered too # error prone. Therefore we use E2E tests that match the Kubernetes # version that is getting tested. -# -# However, for 1.13.x testing we have to use the E2E tests from 1.14 -# because 1.13 didn't have --storage.testdriver yet, so for that (and only -# that version) we have to define alpha tests differently. -configvar CSI_PROW_E2E_ALPHA_1_13 '\[Feature: \[Testpattern:.Dynamic.PV..block.volmode.\] should.create.and.delete.block.persistent.volumes' "alpha tests for Kubernetes 1.13" # Raw block was an alpha feature in 1.13. configvar CSI_PROW_E2E_ALPHA_LATEST '\[Feature:' "alpha tests for Kubernetes >= 1.14" # there's no need to update this, adding a new case for CSI_PROW_E2E for a new Kubernetes is enough configvar CSI_PROW_E2E_ALPHA "$(get_versioned_variable CSI_PROW_E2E_ALPHA "${csi_prow_kubernetes_version_suffix}")" "alpha tests" @@ -312,12 +308,12 @@ configvar CSI_PROW_E2E_ALPHA "$(get_versioned_variable CSI_PROW_E2E_ALPHA "${csi # kubernetes-csi components must be updated, either by disabling # the failing test for "latest" or by updating the test and not running # it anymore for older releases. -configvar CSI_PROW_E2E_ALPHA_GATES_1_13 'VolumeSnapshotDataSource=true,BlockVolume=true,CSIBlockVolume=true' "alpha feature gates for Kubernetes 1.13" configvar CSI_PROW_E2E_ALPHA_GATES_1_14 'VolumeSnapshotDataSource=true,ExpandCSIVolumes=true' "alpha feature gates for Kubernetes 1.14" configvar CSI_PROW_E2E_ALPHA_GATES_1_15 'VolumeSnapshotDataSource=true,ExpandCSIVolumes=true' "alpha feature gates for Kubernetes 1.15" +configvar CSI_PROW_E2E_ALPHA_GATES_1_16 'VolumeSnapshotDataSource=true' "alpha feature gates for Kubernetes 1.16" # TODO: add new CSI_PROW_ALPHA_GATES_xxx entry for future Kubernetes releases and # add new gates to CSI_PROW_E2E_ALPHA_GATES_LATEST. -configvar CSI_PROW_E2E_ALPHA_GATES_LATEST 'VolumeSnapshotDataSource=true,ExpandCSIVolumes=true' "alpha feature gates for latest Kubernetes" +configvar CSI_PROW_E2E_ALPHA_GATES_LATEST 'VolumeSnapshotDataSource=true' "alpha feature gates for latest Kubernetes" configvar CSI_PROW_E2E_ALPHA_GATES "$(get_versioned_variable CSI_PROW_E2E_ALPHA_GATES "${csi_prow_kubernetes_version_suffix}")" "alpha E2E feature gates" # Some tests are known to be unusable in a KinD cluster. For example, @@ -723,22 +719,6 @@ install_sanity () ( run_with_go "${CSI_PROW_GO_VERSION_SANITY}" go test -c -o "${CSI_PROW_WORK}/csi-sanity" "${CSI_PROW_SANITY_IMPORT_PATH}/cmd/csi-sanity" || die "building csi-sanity failed" ) -# Whether the hostpath driver supports raw block devices depends on which version -# we are testing. It would be much nicer if we could determine that by querying the -# installed driver's capabilities instead of having to do a version check. -hostpath_supports_block () { - local result - result="$(docker exec csi-prow-control-plane docker image ls --format='{{.Repository}} {{.Tag}} {{.ID}}' | grep hostpath | while read -r repo tag id; do - if [ "$tag" == "v1.0.1" ]; then - # Old version because the revision label is missing: didn't have support yet. - echo "false" - return - fi - done)" - # If not set, then it must be a newer driver with support. - echo "${result:-true}" -} - # The default implementation of this function generates a external # driver test configuration for the hostpath driver. # @@ -755,12 +735,14 @@ SnapshotClass: DriverInfo: Name: ${CSI_PROW_HOSTPATH_DRIVER_NAME} Capabilities: - block: $(hostpath_supports_block) + block: true persistence: true dataSource: true multipods: true nodeExpansion: true controllerExpansion: true + snapshotDataSource: true + singleNodeVolume: true EOF } From 5e773d2db08fa624d3b9352dd577f885efb7a2b2 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Thu, 31 Oct 2019 08:43:31 +0100 Subject: [PATCH 060/107] update CI to use Go 1.13.3 This is the latest release. Updating is useful to ensure that we have all of the latest fixes and enhancements. --- travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/travis.yml b/travis.yml index a713bed..494c114 100644 --- a/travis.yml +++ b/travis.yml @@ -6,7 +6,7 @@ git: depth: false matrix: include: - - go: 1.12.4 + - go: 1.13.3 before_script: - mkdir -p bin - wget https://github.com/golang/dep/releases/download/v0.5.1/dep-linux-amd64 -O bin/dep From c8a1c4af933311a7e63765cd2b64ca45a0fb7dba Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Thu, 31 Oct 2019 11:49:23 +0100 Subject: [PATCH 061/107] better handling of Go version Some operations are sensitive to the version of Go that is used. In the past, formatting of source differed depending on the version. Right now it is the content of the vendor directory which changes when switch back and forth between 1.12 and 1.13. We don't want to impose a certain workflow on developers, like forcing all invocations of Go to run inside a container. If developers want that, they can set up their development environment accordingly. But we should warn about this aspect to raise awareness. "make" invocations which involve Go now compare against the projects Go version (specified in travis.yml) once at the beginning. This is only a warning because we don't know which future version will be compatible with the project. Vendor directory handling gets updated, too: verification is now a separate script (became too complex for make) and there is a corresponding "update-vendor.sh". In contrast to verification, updating vendor is not integrated into make and thus itself invokes the go version check. --- build.make | 50 ++++++++---------------------------- update-vendor.sh | 23 +++++++++++++++++ verify-go-version.sh | 51 +++++++++++++++++++++++++++++++++++++ verify-vendor.sh | 60 ++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 145 insertions(+), 39 deletions(-) create mode 100755 update-vendor.sh create mode 100755 verify-go-version.sh create mode 100755 verify-vendor.sh diff --git a/build.make b/build.make index cbf6d45..6b89f5b 100644 --- a/build.make +++ b/build.make @@ -62,7 +62,7 @@ ARCH := $(if $(GOARCH),$(GOARCH),$(shell go env GOARCH)) # Specific packages can be excluded from each of the tests below by setting the *_FILTER_CMD variables # to something like "| grep -v 'github.com/kubernetes-csi/project/pkg/foobar'". See usage below. -build-%: +build-%: check-go-version-go mkdir -p bin CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$* ./cmd/$* if [ "$$ARCH" = "amd64" ]; then \ @@ -97,7 +97,7 @@ push: $(CMDS:%=push-%) clean: -rm -rf bin -test: +test: check-go-version-go .PHONY: test-go test: test-go @@ -154,43 +154,7 @@ test-fmt: test: test-vendor test-vendor: @ echo; echo "### $@:" - @ if [ -f Gopkg.toml ]; then \ - echo "Repo uses 'dep' for vendoring."; \ - case "$$(dep version 2>/dev/null | grep 'version *:')" in \ - *v0.[56789]*) dep check && echo "vendor up-to-date" || false;; \ - *) echo "skipping check, dep >= 0.5 required";; \ - esac; \ - elif [ -f go.mod ]; then \ - echo "Repo uses 'go mod'."; \ - if [ "$${JOB_NAME}" ] && \ - ( [ "$${JOB_TYPE}" != "presubmit" ] || \ - [ $$( (git diff "${PULL_BASE_SHA}..HEAD" -- go.mod go.sum vendor release-tools; \ - git diff "${PULL_BASE_SHA}..HEAD" | grep -e '^@@.*@@ import (' -e '^[+-]import') | \ - wc -l) -eq 0 ] ); then \ - echo "Skipping vendor check because the Prow pre-submit job does not affect dependencies."; \ - elif ! GO111MODULE=on go mod tidy; then \ - echo "ERROR: vendor check failed."; \ - false; \ - elif [ $$(git status --porcelain -- go.mod go.sum | wc -l) -gt 0 ]; then \ - echo "ERROR: go module files *not* up-to-date, they did get modified by 'GO111MODULE=on go mod tidy':"; \ - git diff -- go.mod go.sum; \ - false; \ - elif [ -d vendor ]; then \ - if ! GO111MODULE=on go mod vendor; then \ - echo "ERROR: vendor check failed."; \ - false; \ - elif [ $$(git status --porcelain -- vendor | wc -l) -gt 0 ]; then \ - echo "ERROR: vendor directory *not* up-to-date, it did get modified by 'GO111MODULE=on go mod vendor':"; \ - git status -- vendor; \ - git diff -- vendor; \ - false; \ - else \ - echo "Go dependencies and vendor directory up-to-date."; \ - fi; \ - else \ - echo "Go dependencies up-to-date."; \ - fi; \ - fi + @ ./release-tools/verify-vendor.sh .PHONY: test-subtree test: test-subtree @@ -216,3 +180,11 @@ test-shellcheck: ./release-tools/verify-shellcheck.sh "$$dir" || ret=1; \ done; \ exit $$ret + +# Targets in the makefile can depend on check-go-version- +# to trigger a warning if the x.y version of that binary does not match +# what the project uses. Make ensures that this is only checked once per +# invocation. +.PHONY: check-go-version-% +check-go-version-%: + ./release-tools/verify-go-version.sh "$*" diff --git a/update-vendor.sh b/update-vendor.sh new file mode 100755 index 0000000..6f4c27a --- /dev/null +++ b/update-vendor.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash + +# Copyright 2019 The Kubernetes Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +if [ -f Gopkg.toml ]; then + echo "Repo uses 'dep' for vendoring." + (set -x; dep ensure) +elif [ -f go.mod ]; then + release-tools/verify-go-version.sh "go" + (set -x; env GO111MODULE=on go mod tidy && env GO111MODULE=on go mod vendor) +fi diff --git a/verify-go-version.sh b/verify-go-version.sh new file mode 100755 index 0000000..f242e76 --- /dev/null +++ b/verify-go-version.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash + +# Copyright 2019 The Kubernetes Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +GO="$1" + +if [ ! "$GO" ]; then + echo >&2 "usage: $0 " + exit 1 +fi + +die () { + echo "ERROR: $*" + exit 1 +} + +version=$("$GO" version) || die "determining version of $GO failed" +# shellcheck disable=SC2001 +majorminor=$(echo "$version" | sed -e 's/.*go\([0-9]*\)\.\([0-9]*\).*/\1.\2/') +# shellcheck disable=SC2001 +expected=$(grep "^ *- go:" "release-tools/travis.yml" | sed -e 's/.*go: *\([0-9]*\)\.\([0-9]*\).*/\1.\2/') + +if [ "$majorminor" != "$expected" ]; then + cat >&2 </dev/null | grep 'version *:')" in + *v0.[56789]*) + if dep check; then + echo "vendor up-to-date" + else + exit 1 + fi + ;; + *) echo "skipping check, dep >= 0.5 required";; + esac +elif [ -f go.mod ]; then + echo "Repo uses 'go mod'." + # shellcheck disable=SC2235 + if [ "${JOB_NAME}" ] && + ( [ "${JOB_TYPE}" != "presubmit" ] || + [ "$( (git diff "${PULL_BASE_SHA}..HEAD" -- go.mod go.sum vendor release-tools; + git diff "${PULL_BASE_SHA}..HEAD" | grep -e '^@@.*@@ import (' -e '^[+-]import') | + wc -l)" -eq 0 ] ); then + echo "Skipping vendor check because the Prow pre-submit job does not affect dependencies." + elif ! (set -x; env GO111MODULE=on go mod tidy); then + echo "ERROR: vendor check failed." + exit 1 + elif [ "$(git status --porcelain -- go.mod go.sum | wc -l)" -gt 0 ]; then + echo "ERROR: go module files *not* up-to-date, they did get modified by 'GO111MODULE=on go mod tidy':"; + git diff -- go.mod go.sum + exit 1 + elif [ -d vendor ]; then + if ! (set -x; env GO111MODULE=on go mod vendor); then + echo "ERROR: vendor check failed." + exit 1 + elif [ "$(git status --porcelain -- vendor | wc -l)" -gt 0 ]; then + echo "ERROR: vendor directory *not* up-to-date, it did get modified by 'GO111MODULE=on go mod vendor':" + git status -- vendor + git diff -- vendor + exit 1 + else + echo "Go dependencies and vendor directory up-to-date." + fi + else + echo "Go dependencies up-to-date." + fi +fi From 23df4aef51ee82896bc8326a693c3fdbd0baa07f Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Mon, 4 Nov 2019 11:16:38 +0100 Subject: [PATCH 062/107] prow.sh: use vendor directory if available This avoids dependencies on the Go module cache or the upstream code hosting. --- build.make | 11 +++++++---- prow.sh | 12 +++++++++--- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/build.make b/build.make index 6b89f5b..1b6f35f 100644 --- a/build.make +++ b/build.make @@ -22,6 +22,9 @@ # including build.make. REGISTRY_NAME=quay.io/k8scsi +# Can be set to -mod=vendor to ensure that the "vendor" directory is used. +GOFLAGS_VENDOR= + # Revision that gets built into each binary via the main.version # string. Uses the `git describe` output based on the most recent # version tag with a short revision suffix or, if nothing has been @@ -64,9 +67,9 @@ ARCH := $(if $(GOARCH),$(GOARCH),$(shell go env GOARCH)) build-%: check-go-version-go mkdir -p bin - CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$* ./cmd/$* + CGO_ENABLED=0 GOOS=linux go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$* ./cmd/$* if [ "$$ARCH" = "amd64" ]; then \ - CGO_ENABLED=0 GOOS=windows go build -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$*.exe ./cmd/$* ; \ + CGO_ENABLED=0 GOOS=windows go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$*.exe ./cmd/$* ; \ fi container-%: build-% @@ -103,13 +106,13 @@ test: check-go-version-go test: test-go test-go: @ echo; echo "### $@:" - go test `go list ./... | grep -v -e 'vendor' -e '/test/e2e$$' $(TEST_GO_FILTER_CMD)` $(TESTARGS) + go test $(GOFLAGS_VENDOR) `go list $(GOFLAGS_VENDOR) ./... | grep -v -e 'vendor' -e '/test/e2e$$' $(TEST_GO_FILTER_CMD)` $(TESTARGS) .PHONY: test-vet test: test-vet test-vet: @ echo; echo "### $@:" - go vet `go list ./... | grep -v vendor $(TEST_VET_FILTER_CMD)` + go test $(GOFLAGS_VENDOR) `go list $(GOFLAGS_VENDOR) ./... | grep -v vendor $(TEST_VET_FILTER_CMD)` .PHONY: test-fmt test: test-fmt diff --git a/prow.sh b/prow.sh index e3308e7..fe68bb7 100755 --- a/prow.sh +++ b/prow.sh @@ -85,6 +85,12 @@ get_versioned_variable () { echo "$value" } +# If we have a vendor directory, then use it. We must be careful to only +# use this for "make" invocations inside the project's repo itself because +# setting it globally can break other go usages (like "go get " +# which is disabled with GOFLAGS=-mod=vendor). +configvar GOFLAGS_VENDOR "$( [ -d vendor ] && echo '-mod=vendor' )" "Go flags for using the vendor directory" + # Go versions can be specified seperately for different tasks # If the pre-installed Go is missing or a different # version, the required version here will get installed @@ -928,7 +934,7 @@ main () { images= if ${CSI_PROW_BUILD_JOB}; then # A successful build is required for testing. - run_with_go "${CSI_PROW_GO_VERSION_BUILD}" make all || die "'make all' failed" + run_with_go "${CSI_PROW_GO_VERSION_BUILD}" make all "GOFLAGS_VENDOR=${GOFLAGS_VENDOR}" || die "'make all' failed" # We don't want test failures to prevent E2E testing below, because the failure # might have been minor or unavoidable, for example when experimenting with # changes in "release-tools" in a PR (that fails the "is release-tools unmodified" @@ -938,13 +944,13 @@ main () { warn "installing 'dep' failed, cannot test vendoring" ret=1 fi - if ! run_with_go "${CSI_PROW_GO_VERSION_BUILD}" make -k test 2>&1 | make_test_to_junit; then + if ! run_with_go "${CSI_PROW_GO_VERSION_BUILD}" make -k test "GOFLAGS_VENDOR=${GOFLAGS_VENDOR}" 2>&1 | make_test_to_junit; then warn "'make test' failed, proceeding anyway" ret=1 fi fi # Required for E2E testing. - run_with_go "${CSI_PROW_GO_VERSION_BUILD}" make container || die "'make container' failed" + run_with_go "${CSI_PROW_GO_VERSION_BUILD}" make container "GOFLAGS_VENDOR=${GOFLAGS_VENDOR}" || die "'make container' failed" fi if tests_need_kind; then From 806784565969caa302d40f98f6fb6dd3dafb39ba Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Mon, 4 Nov 2019 15:40:31 +0100 Subject: [PATCH 063/107] travis.yml: also use vendor directory --- travis.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/travis.yml b/travis.yml index 494c114..bfd7647 100644 --- a/travis.yml +++ b/travis.yml @@ -13,9 +13,9 @@ before_script: - chmod u+x bin/dep - export PATH=$PWD/bin:$PATH script: -- make -k all test +- make -k all test GOFLAGS_VENDOR=$( [ -d vendor ] && echo '-mod=vendor' ) after_success: - if [ "${TRAVIS_PULL_REQUEST}" == "false" ]; then docker login -u "${DOCKER_USERNAME}" -p "${DOCKER_PASSWORD}" quay.io; - make push; + make push GOFLAGS_VENDOR=$( [ -d vendor ] && echo '-mod=vendor' ); fi From f41c1351a38ad936cbdec298fff3f49b8e2e8613 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Tue, 5 Nov 2019 16:40:29 +0100 Subject: [PATCH 064/107] prow.sh: also log output of system containers Depending on the error, those logs are needed to debug failures. --- prow.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/prow.sh b/prow.sh index e3308e7..0777ae3 100755 --- a/prow.sh +++ b/prow.sh @@ -677,16 +677,16 @@ EOF } -# Gets logs of all containers in the default namespace. When passed -f, kubectl will +# Gets logs of all containers in all namespaces. When passed -f, kubectl will # keep running and capture new output. Prints the pid of all background processes. # The caller must kill (when using -f) and/or wait for them. # # May be called multiple times and thus appends. start_loggers () { - kubectl get pods -o go-template --template='{{range .items}}{{.metadata.name}} {{range .spec.containers}}{{.name}} {{end}}{{"\n"}}{{end}}' | while read -r pod containers; do + kubectl get pods --all-namespaces -o go-template --template='{{range .items}}{{.metadata.namespace}} {{.metadata.name}} {{range .spec.containers}}{{.name}} {{end}}{{"\n"}}{{end}}' | while read -r namespace pod containers; do for container in $containers; do - mkdir -p "${ARTIFACTS}/$pod" - kubectl logs "$@" "$pod" "$container" >>"${ARTIFACTS}/$pod/$container.log" & + mkdir -p "${ARTIFACTS}/$namespace/$pod" + kubectl logs -n "$namespace" "$@" "$pod" "$container" >>"${ARTIFACTS}/$namespace/$pod/$container.log" & echo "$!" done done From 83a4ef15dfda6403862437f8acf3606f255e888a Mon Sep 17 00:00:00 2001 From: Peeyush Gupta Date: Tue, 19 Nov 2019 08:58:58 -0500 Subject: [PATCH 065/107] Adding build for ppc64le --- build.make | 1 + 1 file changed, 1 insertion(+) diff --git a/build.make b/build.make index 1b6f35f..7075a37 100644 --- a/build.make +++ b/build.make @@ -70,6 +70,7 @@ build-%: check-go-version-go CGO_ENABLED=0 GOOS=linux go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$* ./cmd/$* if [ "$$ARCH" = "amd64" ]; then \ CGO_ENABLED=0 GOOS=windows go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$*.exe ./cmd/$* ; \ + CGO_ENABLED=0 GOOS=linux GOARCH=ppc64le go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$*-ppc64le ./cmd/$* fi container-%: build-% From 1eaaaa1cb49713e8e4af87d009c24d310cf66260 Mon Sep 17 00:00:00 2001 From: Mucahit Kurt Date: Sat, 16 Nov 2019 06:23:08 +0300 Subject: [PATCH 066/107] Delete kind cluster after tests run. Inside a real Prow job it is better to clean up at runtime instead of leaving that to the Prow job cleanup code because the later sometimes times out. Signed-off-by: Mucahit Kurt --- prow.sh | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/prow.sh b/prow.sh index 3b9621a..8b431c5 100755 --- a/prow.sh +++ b/prow.sh @@ -580,6 +580,19 @@ EOF export KUBECONFIG } +# Deletes kind cluster inside a prow job +delete_cluster_inside_prow_job() { + # Inside a real Prow job it is better to clean up at runtime + # instead of leaving that to the Prow job cleanup code + # because the later sometimes times out (https://github.com/kubernetes-csi/csi-release-tools/issues/24#issuecomment-554765872). + if [ "$JOB_NAME" ]; then + if kind get clusters | grep -q csi-prow; then + run kind delete cluster --name=csi-prow || die "kind delete failed" + fi + unset KUBECONFIG + fi +} + # Looks for the deployment as specified by CSI_PROW_DEPLOYMENT and CSI_PROW_KUBERNETES_VERSION # in the given directory. find_deployment () { @@ -1017,6 +1030,7 @@ main () { fi fi fi + delete_cluster_inside_prow_job fi if tests_need_alpha_cluster && [ "${CSI_PROW_E2E_ALPHA_GATES}" ]; then @@ -1047,6 +1061,7 @@ main () { fi fi fi + delete_cluster_inside_prow_job fi fi From 003c14b2d4ae3b1463db5e5b3ff91f39b03f5ba8 Mon Sep 17 00:00:00 2001 From: Grant Griffiths Date: Mon, 11 Nov 2019 23:49:42 -0800 Subject: [PATCH 067/107] Add snapshotter CRDs after cluster setup Signed-off-by: Grant Griffiths --- prow.sh | 80 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/prow.sh b/prow.sh index 3b9621a..bc9c9f0 100755 --- a/prow.sh +++ b/prow.sh @@ -322,6 +322,9 @@ configvar CSI_PROW_E2E_ALPHA_GATES_1_16 'VolumeSnapshotDataSource=true' "alpha f configvar CSI_PROW_E2E_ALPHA_GATES_LATEST 'VolumeSnapshotDataSource=true' "alpha feature gates for latest Kubernetes" configvar CSI_PROW_E2E_ALPHA_GATES "$(get_versioned_variable CSI_PROW_E2E_ALPHA_GATES "${csi_prow_kubernetes_version_suffix}")" "alpha E2E feature gates" +# Which external-snapshotter tag to use for the snapshotter CRD and snapshot-controller deployment +configvar CSI_SNAPSHOTTER_VERSION 'v2.0.0-rc4' "external-snapshotter version tag" + # Some tests are known to be unusable in a KinD cluster. For example, # stopping kubelet with "ssh systemctl stop kubelet" simply # doesn't work. Such tests should be written in a way that they verify @@ -657,6 +660,59 @@ install_hostpath () { fi } +# Installs all nessesary snapshotter CRDs +install_snapshot_crds() { + # Wait until volumesnapshot CRDs are in place. + CRD_BASE_DIR="https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${CSI_SNAPSHOTTER_VERSION}/config/crd" + kubectl apply -f "${CRD_BASE_DIR}/snapshot.storage.k8s.io_volumesnapshotclasses.yaml" --validate=false + kubectl apply -f "${CRD_BASE_DIR}/snapshot.storage.k8s.io_volumesnapshots.yaml" --validate=false + kubectl apply -f "${CRD_BASE_DIR}/snapshot.storage.k8s.io_volumesnapshotcontents.yaml" --validate=false + cnt=0 + until kubectl get volumesnapshotclasses.snapshot.storage.k8s.io \ + && kubectl get volumesnapshots.snapshot.storage.k8s.io \ + && kubectl get volumesnapshotcontents.snapshot.storage.k8s.io; do + if [ $cnt -gt 30 ]; then + echo >&2 "ERROR: snapshot CRDs not ready after over 1 min" + exit 1 + fi + echo "$(date +%H:%M:%S)" "waiting for snapshot CRDs, attempt #$cnt" + cnt=$((cnt + 1)) + sleep 2 + done +} + +# Install snapshot controller and associated RBAC, retrying until the pod is running. +install_snapshot_controller() { + kubectl apply -f "https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${CSI_SNAPSHOTTER_VERSION}/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml" + cnt=0 + until kubectl get clusterrolebinding snapshot-controller-role; do + if [ $cnt -gt 30 ]; then + echo "Cluster role bindings:" + kubectl describe clusterrolebinding + echo >&2 "ERROR: snapshot controller RBAC not ready after over 5 min" + exit 1 + fi + echo "$(date +%H:%M:%S)" "waiting for snapshot RBAC setup complete, attempt #$cnt" + cnt=$((cnt + 1)) + sleep 10 + done + + + kubectl apply -f "https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${CSI_SNAPSHOTTER_VERSION}/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml" + cnt=0 + until kubectl get statefulset snapshot-controller | grep snapshot-controller | grep "1/1"; do + if [ $cnt -gt 30 ]; then + echo "Running statefulsets:" + kubectl describe statefulsets + echo >&2 "ERROR: snapshot controller not ready after over 5 min" + exit 1 + fi + echo "$(date +%H:%M:%S)" "waiting for snapshot controller deployment to complete, attempt #$cnt" + cnt=$((cnt + 1)) + sleep 10 + done +} + # collect logs and cluster status (like the version of all components, Kubernetes version, test version) collect_cluster_info () { cat < Date: Sat, 30 Nov 2019 00:29:00 +0530 Subject: [PATCH 068/107] Use kind v0.6.0 kind v0.6.0 appends the kubeconfig with the default config at ~/.kube/config. --- prow.sh | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/prow.sh b/prow.sh index d719e98..b882306 100755 --- a/prow.sh +++ b/prow.sh @@ -107,8 +107,7 @@ configvar CSI_PROW_GO_VERSION_GINKGO "${CSI_PROW_GO_VERSION_BUILD}" "Go version # kind version to use. If the pre-installed version is different, # the desired version is downloaded from https://github.com/kubernetes-sigs/kind/releases/download/ # (if available), otherwise it is built from source. -# TODO: https://github.com/kubernetes-csi/csi-release-tools/issues/39 -configvar CSI_PROW_KIND_VERSION "86bc23d84ac12dcb56a0528890736e2c347c2dc3" "kind" +configvar CSI_PROW_KIND_VERSION "v0.6.0" "kind" # ginkgo test runner version to use. If the pre-installed version is # different, the desired version is built from source. @@ -579,8 +578,7 @@ EOF die "Cluster creation failed again, giving up. See the 'kind-cluster' artifact directory for additional logs." fi fi - KUBECONFIG="$(kind get kubeconfig-path --name=csi-prow)" - export KUBECONFIG + export KUBECONFIG="${HOME}/.kube/config" } # Deletes kind cluster inside a prow job From 9a7a685ee169d669a0182532315aeecd8a8715dc Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Mon, 2 Dec 2019 17:45:57 -0800 Subject: [PATCH 069/107] Create a kind cluster with two worker nodes so that the topology feature can be tested. Test cases that test accessing volumes from multiple nodes need to be skipped --- prow.sh | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/prow.sh b/prow.sh index b882306..815bc9d 100755 --- a/prow.sh +++ b/prow.sh @@ -330,7 +330,11 @@ configvar CSI_SNAPSHOTTER_VERSION 'v2.0.0-rc4' "external-snapshotter version tag # whether they can run with the current cluster provider, but until # they are, we filter them out by name. Like the other test selection # variables, this is again a space separated list of regular expressions. -configvar CSI_PROW_E2E_SKIP 'Disruptive' "tests that need to be skipped" +# +# "different node" test skips can be removed once +# https://github.com/kubernetes/kubernetes/pull/82678 has been backported +# to all the K8s versions we test against +configvar CSI_PROW_E2E_SKIP 'Disruptive|different\s+node' "tests that need to be skipped" # This is the directory for additional result files. Usually set by Prow, but # if not (for example, when invoking manually) it defaults to the work directory. @@ -526,6 +530,7 @@ apiVersion: kind.sigs.k8s.io/v1alpha3 nodes: - role: control-plane - role: worker +- role: worker EOF # kubeadm has API dependencies between apiVersion and Kubernetes version @@ -840,10 +845,6 @@ run_e2e () ( install_e2e || die "building e2e.test failed" install_ginkgo || die "installing ginkgo failed" - # TODO (?): multi-node cluster (depends on https://github.com/kubernetes-csi/csi-driver-host-path/pull/14). - # When running on a multi-node cluster, we need to figure out where the - # hostpath driver was deployed and set ClientNodeName accordingly. - generate_test_driver >"${CSI_PROW_WORK}/test-driver.yaml" || die "generating test-driver.yaml failed" # Rename, merge and filter JUnit files. Necessary in case that we run the E2E suite again From 53888ae7d59b862bfded46b883570c98f5fc5fff Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Tue, 3 Dec 2019 18:18:38 -0800 Subject: [PATCH 070/107] Improve README by adding an explicit Kubernetes dependency section --- README.md | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index cc40f16..08c8288 100644 --- a/README.md +++ b/README.md @@ -141,17 +141,6 @@ The `vendor` directory is optional. It is still present in projects because it avoids downloading sources during CI builds. If this is no longer deemed necessary, then a project can also remove the directory. -When using packages that are part of the Kubernetes source code, the -commands above are not enough because the [lack of semantic -versioning](https://github.com/kubernetes/kubernetes/issues/72638) -prevents `go mod` from finding newer releases. Importing directly from -`kubernetes/kubernetes` also needs `replace` statements to override -the fake `v0.0.0` versions -(https://github.com/kubernetes/kubernetes/issues/79384). The -`go-get-kubernetes.sh` script can be used to update all packages in -lockstep to a different Kubernetes version. It takes a single version -number like "1.16.0". - Conversion of a repository that uses `dep` to `go mod` can be done with: GO111MODULE=on go mod init @@ -160,3 +149,18 @@ Conversion of a repository that uses `dep` to `go mod` can be done with: GO111MODULE=on go mod vendor git rm -f Gopkg.toml Gopkg.lock git add go.mod go.sum vendor + +### Updating Kubernetes dependencies + +When using packages that are part of the Kubernetes source code, the +commands above are not enough because the [lack of semantic +versioning](https://github.com/kubernetes/kubernetes/issues/72638) +prevents `go mod` from finding newer releases. Importing directly from +`kubernetes/kubernetes` also needs `replace` statements to override +the fake `v0.0.0` versions +(https://github.com/kubernetes/kubernetes/issues/79384). The +`go-get-kubernetes.sh` script can be used to update all packages in +lockstep to a different Kubernetes version. Example usage: +``` +$ ./release-tools/go-get-kubernetes.sh 1.16.4 +``` From 4ad69492c97834b52d74d0b67b6f47b0590e4be1 Mon Sep 17 00:00:00 2001 From: Grant Griffiths Date: Tue, 3 Dec 2019 23:48:29 -0800 Subject: [PATCH 071/107] Improve snapshot pod running checks and improve version_gt Signed-off-by: Grant Griffiths --- prow.sh | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-) diff --git a/prow.sh b/prow.sh index d719e98..27c7051 100755 --- a/prow.sh +++ b/prow.sh @@ -713,10 +713,11 @@ install_snapshot_controller() { kubectl apply -f "https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${CSI_SNAPSHOTTER_VERSION}/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml" cnt=0 - until kubectl get statefulset snapshot-controller | grep snapshot-controller | grep "1/1"; do + expected_running_pods=$(curl https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/"${CSI_SNAPSHOTTER_VERSION}"/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml | grep replicas | cut -d ':' -f 2-) + while [ "$(kubectl get pods -l app=snapshot-controller | grep 'Running' -c)" -lt "$expected_running_pods" ]; do if [ $cnt -gt 30 ]; then - echo "Running statefulsets:" - kubectl describe statefulsets + echo "snapshot-controller pod status:" + kubectl describe pods -l app=snapshot-controller echo >&2 "ERROR: snapshot controller not ready after over 5 min" exit 1 fi @@ -996,8 +997,30 @@ make_test_to_junit () { fi } +# version_gt returns true if arg1 is greater than arg2. +# +# This function expects versions to be one of the following formats: +# X.Y.Z, release-X.Y.Z, vX.Y.Z +# +# where X,Y, and Z are any number. +# +# Partial versions (1.2, release-1.2) work as well. +# The follow substrings are stripped before version comparison: +# - "v" +# - "release-" +# +# Usage: +# version_gt release-1.3 v1.2.0 (returns true) +# version_gt v1.1.1 v1.2.0 (returns false) +# version_gt 1.1.1 v1.2.0 (returns false) +# version_gt 1.3.1 v1.2.0 (returns true) +# version_gt 1.1.1 release-1.2.0 (returns false) +# version_gt 1.2.0 1.2.2 (returns false) function version_gt() { - test "$(printf '%s\n' "$@" | sort -V | head -n 1)" != "$1"; + versions=$(for ver in "$@"; do ver=${ver#release-}; echo "${ver#v}"; done) + greaterVersion=${1#"release-"}; + greaterVersion=${greaterVersion#"v"}; + test "$(printf '%s' "$versions" | sort -V | head -n 1)" != "$greaterVersion" } main () { From a4e629966848d0097461f49f8aab0dfeb68d7a68 Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Wed, 4 Dec 2019 14:12:50 -0800 Subject: [PATCH 072/107] fix syntax for ppc64le build --- build.make | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.make b/build.make index 7075a37..a9b9d25 100644 --- a/build.make +++ b/build.make @@ -70,7 +70,7 @@ build-%: check-go-version-go CGO_ENABLED=0 GOOS=linux go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$* ./cmd/$* if [ "$$ARCH" = "amd64" ]; then \ CGO_ENABLED=0 GOOS=windows go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$*.exe ./cmd/$* ; \ - CGO_ENABLED=0 GOOS=linux GOARCH=ppc64le go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$*-ppc64le ./cmd/$* + CGO_ENABLED=0 GOOS=linux GOARCH=ppc64le go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$*-ppc64le ./cmd/$* ; \ fi container-%: build-% From b98b2aed087f4e768db673a9d43dff4c7abd1482 Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Mon, 16 Dec 2019 19:13:38 -0800 Subject: [PATCH 073/107] Enable snapshot tests in 1.17 to be run in non-alpha jobs. This requires adding one more parallel e2e test run with a special focus flag because snapshot tests are still guarded with a "[Feature:VolumeSnapshotDataSource]" tag. The setting that skips all tests with "[Feature:.*]" has to be removed because it overrides the focus. We don't have serial snapshot tests yet. This needs to be modified again if we add any in the future. --- prow.sh | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/prow.sh b/prow.sh index 9bfdc2e..c160f88 100755 --- a/prow.sh +++ b/prow.sh @@ -132,7 +132,7 @@ configvar CSI_PROW_BUILD_JOB true "building code in repo enabled" # use the same settings as for "latest" Kubernetes. This works # as long as there are no breaking changes in Kubernetes, like # deprecating or changing the implementation of an alpha feature. -configvar CSI_PROW_KUBERNETES_VERSION 1.15.3 "Kubernetes" +configvar CSI_PROW_KUBERNETES_VERSION 1.17.0 "Kubernetes" # This is a hack to workaround the issue that each version # of kind currently only supports specific patch versions of @@ -142,7 +142,6 @@ configvar CSI_PROW_KUBERNETES_VERSION 1.15.3 "Kubernetes" # # If the version is prefixed with "release-", then nothing # is overridden. -override_k8s_version "1.14.6" override_k8s_version "1.15.3" # CSI_PROW_KUBERNETES_VERSION reduced to first two version numbers and @@ -206,9 +205,9 @@ configvar CSI_PROW_HOSTPATH_CANARY "" "hostpath image" # # CSI_PROW_E2E_REPO=none disables E2E testing. # TOOO: remove versioned variables and make e2e version match k8s version -configvar CSI_PROW_E2E_VERSION_1_14 v1.14.0 "E2E version for Kubernetes 1.14.x" configvar CSI_PROW_E2E_VERSION_1_15 v1.15.0 "E2E version for Kubernetes 1.15.x" configvar CSI_PROW_E2E_VERSION_1_16 v1.16.0 "E2E version for Kubernetes 1.16.x" +configvar CSI_PROW_E2E_VERSION_1_17 v1.17.0 "E2E version for Kubernetes 1.17.x" # TODO: add new CSI_PROW_E2E_VERSION entry for future Kubernetes releases configvar CSI_PROW_E2E_VERSION_LATEST master "E2E version for Kubernetes master" # testing against Kubernetes master is already tracking a moving target, so we might as well use a moving E2E version configvar CSI_PROW_E2E_REPO_LATEST https://github.com/kubernetes/kubernetes "E2E repo for Kubernetes >= 1.13.x" # currently the same for all versions @@ -278,6 +277,14 @@ tests_need_alpha_cluster () { tests_enabled "parallel-alpha" "serial-alpha" } +# Regex for non-alpha, feature-tagged tests that should be run. +# +# Starting with 1.17, snapshots is beta, but the E2E tests still have the +# [Feature:] tag. They need to be explicitly enabled. +configvar CSI_PROW_E2E_FOCUS_1_15 '^' "non-alpha, feature-tagged tests for Kubernetes = 1.15" # no tests to run, match nothing +configvar CSI_PROW_E2E_FOCUS_1_16 '^' "non-alpha, feature-tagged tests for Kubernetes = 1.16" # no tests to run, match nothing +configvar CSI_PROW_E2E_FOCUS_LATEST '\[Feature:VolumeSnapshotDataSource\]' "non-alpha, feature-tagged tests for Kubernetes >= 1.17" +configvar CSI_PROW_E2E_FOCUS "$(get_versioned_variable CSI_PROW_E2E_FOCUS "${csi_prow_kubernetes_version_suffix}")" "non-alpha, feature-tagged tests" # Serial vs. parallel is always determined by these regular expressions. # Individual regular expressions are seperated by spaces for readability @@ -313,12 +320,11 @@ configvar CSI_PROW_E2E_ALPHA "$(get_versioned_variable CSI_PROW_E2E_ALPHA "${csi # kubernetes-csi components must be updated, either by disabling # the failing test for "latest" or by updating the test and not running # it anymore for older releases. -configvar CSI_PROW_E2E_ALPHA_GATES_1_14 'VolumeSnapshotDataSource=true,ExpandCSIVolumes=true' "alpha feature gates for Kubernetes 1.14" configvar CSI_PROW_E2E_ALPHA_GATES_1_15 'VolumeSnapshotDataSource=true,ExpandCSIVolumes=true' "alpha feature gates for Kubernetes 1.15" configvar CSI_PROW_E2E_ALPHA_GATES_1_16 'VolumeSnapshotDataSource=true' "alpha feature gates for Kubernetes 1.16" # TODO: add new CSI_PROW_ALPHA_GATES_xxx entry for future Kubernetes releases and # add new gates to CSI_PROW_E2E_ALPHA_GATES_LATEST. -configvar CSI_PROW_E2E_ALPHA_GATES_LATEST 'VolumeSnapshotDataSource=true' "alpha feature gates for latest Kubernetes" +configvar CSI_PROW_E2E_ALPHA_GATES_LATEST '' "alpha feature gates for latest Kubernetes" configvar CSI_PROW_E2E_ALPHA_GATES "$(get_versioned_variable CSI_PROW_E2E_ALPHA_GATES "${csi_prow_kubernetes_version_suffix}")" "alpha E2E feature gates" # Which external-snapshotter tag to use for the snapshotter CRD and snapshot-controller deployment @@ -1111,6 +1117,16 @@ main () { warn "E2E parallel failed" ret=1 fi + + # Run tests that are feature tagged, but non-alpha + # Ignore: Double quote to prevent globbing and word splitting. + # shellcheck disable=SC2086 + if ! run_e2e parallel ${CSI_PROW_GINKO_PARALLEL} \ + -focus="External.Storage.*($(regex_join "${CSI_PROW_E2E_FOCUS}"))" \ + -skip="$(regex_join "${CSI_PROW_E2E_SERIAL}")"; then + warn "E2E parallel features failed" + ret=1 + fi fi if tests_enabled "serial"; then From fc80975954a9720a1611fa9a5195cbfc48b64167 Mon Sep 17 00:00:00 2001 From: Grant Griffiths Date: Fri, 20 Dec 2019 16:30:25 -0800 Subject: [PATCH 074/107] Fix version_gt to work with kubernetes prefix Signed-off-by: Grant Griffiths --- prow.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/prow.sh b/prow.sh index c160f88..832da1e 100755 --- a/prow.sh +++ b/prow.sh @@ -1013,6 +1013,7 @@ make_test_to_junit () { # The follow substrings are stripped before version comparison: # - "v" # - "release-" +# - "kubernetes-" # # Usage: # version_gt release-1.3 v1.2.0 (returns true) @@ -1022,8 +1023,9 @@ make_test_to_junit () { # version_gt 1.1.1 release-1.2.0 (returns false) # version_gt 1.2.0 1.2.2 (returns false) function version_gt() { - versions=$(for ver in "$@"; do ver=${ver#release-}; echo "${ver#v}"; done) + versions=$(for ver in "$@"; do ver=${ver#release-}; ver=${ver#kubernetes-}; echo "${ver#v}"; done) greaterVersion=${1#"release-"}; + greaterVersion=${greaterVersion#"kubernetes-"}; greaterVersion=${greaterVersion#"v"}; test "$(printf '%s' "$versions" | sort -V | head -n 1)" != "$greaterVersion" } From af9549b5a11c4849ff7b4a1a42a7b8aea59137c3 Mon Sep 17 00:00:00 2001 From: saad-ali Date: Thu, 2 Jan 2020 14:29:40 -0800 Subject: [PATCH 075/107] Update prow hostpath driver version to 1.3.0-rc2 --- prow.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prow.sh b/prow.sh index 832da1e..ae51947 100755 --- a/prow.sh +++ b/prow.sh @@ -187,7 +187,7 @@ configvar CSI_PROW_WORK "$(mkdir -p "$GOPATH/pkg" && mktemp -d "$GOPATH/pkg/csip # # When no deploy script is found (nothing in `deploy` directory, # CSI_PROW_HOSTPATH_REPO=none), nothing gets deployed. -configvar CSI_PROW_HOSTPATH_VERSION "v1.2.0" "hostpath driver" +configvar CSI_PROW_HOSTPATH_VERSION "v1.3.0-rc2" "hostpath driver" configvar CSI_PROW_HOSTPATH_REPO https://github.com/kubernetes-csi/csi-driver-host-path "hostpath repo" configvar CSI_PROW_DEPLOYMENT "" "deployment" configvar CSI_PROW_HOSTPATH_DRIVER_NAME "hostpath.csi.k8s.io" "the hostpath driver name" From 8b0316c7e4d94a3206cdab0ea2e8d7f7b3be637b Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Thu, 2 Jan 2020 14:33:46 -0800 Subject: [PATCH 076/107] Fix overriding of junit results by using unique names for each e2e run --- prow.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prow.sh b/prow.sh index ae51947..e21a7ba 100755 --- a/prow.sh +++ b/prow.sh @@ -1123,7 +1123,7 @@ main () { # Run tests that are feature tagged, but non-alpha # Ignore: Double quote to prevent globbing and word splitting. # shellcheck disable=SC2086 - if ! run_e2e parallel ${CSI_PROW_GINKO_PARALLEL} \ + if ! run_e2e parallel-features ${CSI_PROW_GINKO_PARALLEL} \ -focus="External.Storage.*($(regex_join "${CSI_PROW_E2E_FOCUS}"))" \ -skip="$(regex_join "${CSI_PROW_E2E_SERIAL}")"; then warn "E2E parallel features failed" From 8191eab6ffe694d11bf1e0d254330287c961dc77 Mon Sep 17 00:00:00 2001 From: Grant Griffiths Date: Tue, 7 Jan 2020 18:09:31 -0800 Subject: [PATCH 077/107] Update snapshotter to version v2.0.0 Signed-off-by: Grant Griffiths --- prow.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prow.sh b/prow.sh index e21a7ba..eb451ef 100755 --- a/prow.sh +++ b/prow.sh @@ -328,7 +328,7 @@ configvar CSI_PROW_E2E_ALPHA_GATES_LATEST '' "alpha feature gates for latest Kub configvar CSI_PROW_E2E_ALPHA_GATES "$(get_versioned_variable CSI_PROW_E2E_ALPHA_GATES "${csi_prow_kubernetes_version_suffix}")" "alpha E2E feature gates" # Which external-snapshotter tag to use for the snapshotter CRD and snapshot-controller deployment -configvar CSI_SNAPSHOTTER_VERSION 'v2.0.0-rc4' "external-snapshotter version tag" +configvar CSI_SNAPSHOTTER_VERSION 'v2.0.0' "external-snapshotter version tag" # Some tests are known to be unusable in a KinD cluster. For example, # stopping kubelet with "ssh systemctl stop kubelet" simply From 6582f2ff3bd584e662035b7da2a90efc52184b1b Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Thu, 9 Jan 2020 17:25:38 -0800 Subject: [PATCH 078/107] Update hostpath driver version to get fix for connection-timeout --- prow.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prow.sh b/prow.sh index e21a7ba..27309a7 100755 --- a/prow.sh +++ b/prow.sh @@ -187,7 +187,7 @@ configvar CSI_PROW_WORK "$(mkdir -p "$GOPATH/pkg" && mktemp -d "$GOPATH/pkg/csip # # When no deploy script is found (nothing in `deploy` directory, # CSI_PROW_HOSTPATH_REPO=none), nothing gets deployed. -configvar CSI_PROW_HOSTPATH_VERSION "v1.3.0-rc2" "hostpath driver" +configvar CSI_PROW_HOSTPATH_VERSION "v1.3.0-rc3" "hostpath driver" configvar CSI_PROW_HOSTPATH_REPO https://github.com/kubernetes-csi/csi-driver-host-path "hostpath repo" configvar CSI_PROW_DEPLOYMENT "" "deployment" configvar CSI_PROW_HOSTPATH_DRIVER_NAME "hostpath.csi.k8s.io" "the hostpath driver name" From ac8a0212b93cad817689029be503839ad1959e21 Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Wed, 11 Dec 2019 17:56:08 -0800 Subject: [PATCH 079/107] Document the process for releasing a new sidecar --- SIDECAR_RELEASE_PROCESS.md | 90 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100644 SIDECAR_RELEASE_PROCESS.md diff --git a/SIDECAR_RELEASE_PROCESS.md b/SIDECAR_RELEASE_PROCESS.md new file mode 100644 index 0000000..91a1e1b --- /dev/null +++ b/SIDECAR_RELEASE_PROCESS.md @@ -0,0 +1,90 @@ +# Sidecar Release Process + +This page describes the process for releasing a kubernetes-csi sidecar. + +## Prerequisites + +The release manager must: + +* Be a member of the kubernetes-csi organization. Open an + [issue](https://github.com/kubernetes/org/issues/new?assignees=&labels=area%2Fgithub-membership&template=membership.md&title=REQUEST%3A+New+membership+for+%3Cyour-GH-handle%3E) in + kubernetes/org to request membership +* Be a top level approver for the repository. To become a top level approver, + the candidate must demonstrate ownership and deep knowledge of the repository + through active maintainence, responding to and fixing issues, reviewing PRs, + test triage. +* Be part of the maintainers or admin group for the repository. admin is a + superset of maintainers, only maintainers level is required for cutting a + release. Membership can be requested by submitting a PR to kubernetes/org. + [Example](https://github.com/kubernetes/org/pull/1467) + +## Updating CI Jobs +Whenever a new Kubernetes minor version is released, our kubernetes-csi CI jobs +must be updated. + +[Our CI jobs](https://k8s-testgrid.appspot.com/sig-storage-csi-ci) have the +naming convention `-on-`. + +1. Jobs should be actively monitored to find and fix failures in sidecars and + infrastructure changes early in the development cycle. Test failures are sent + to kubernetes-sig-storage-test-failures@googlegroups.com. +1. "-on-master" jobs are the closest reflection to the new Kubernetes version. +1. Fixes to our prow.sh CI script can be tested in the [CSI hostpath + repo](https://github.com/kubernetes-csi/csi-driver-host-path) by modifying + [prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/release-tools/prow.sh) + along with any overrides in + [.prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/.prow.sh) + to mirror the failing environment. Once e2e tests are passing (verify-unit tests + will fail), then the prow.sh changes can be submitted to [csi-release-tools](https://github.com/kubernetes-csi/csi-release-tools). +1. Changes can then be updated in all the sidecar repos and hostpath driver repo + by following the [update + instructions](https://github.com/kubernetes-csi/csi-release-tools/blob/master/README.md#sharing-and-updating). +1. New pull and CI jobs are configured by + [gen-jobs.sh](https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-csi/gen-jobs.sh). + New pull jobs that have been unverified should be initially made optional. + [Example](https://github.com/kubernetes/test-infra/pull/15055) +1. Once new pull and CI jobs have been verified, and the new Kubernetes version + is released, we can make the optional jobs required, and also remove the + Kubernetes versions that are no longer supported. + +## Release Process +1. Identify all issues and ongoing PRs that should go into the release, and + drive them to resolution. +1. Download [K8s release notes + generator](https://github.com/kubernetes/release/tree/master/cmd/release-notes) +1. Generate release notes for the release. Replace arguments with the relevant + information. + ``` + GITHUB_TOKEN= ./release-notes --start-sha=0ed6978fd199e3ca10326b82b4b8b8e916211c9b --end-sha=3cb3d2f18ed8cb40371c6d8886edcabd1f27e7b9 \ + --github-org=kubernetes-csi --github-repo=external-attacher -branch=master -output out.md + ``` + * `--start-sha` should point to the last release from the same branch. For + example: + * `1.X-1.0` tag when releasing `1.X.0` + * `1.X.Y-1` tag when releasing `1.X.Y` +1. Compare the generated output to the new commits for the release to check if + any notable change missed a release note. +1. Reword release notes as needed. Make sure to check notes for breaking + changes and deprecations. +1. If release is a new major/minor version, create a new `CHANGELOG-..md` + file. Otherwise, add the release notes to the top of the existing CHANGELOG + file for that minor version. +1. Submit a PR for the CHANGELOG changes. +1. Submit a PR for README changes, in particular, Compatibility, Feature status, + and any other sections that may need updating. +1. Check that all [canary CI + jobs](https://k8s-testgrid.appspot.com/sig-storage-csi-ci) are passing, + and that test coverage is adequate for the changes that are going into the release. +1. Make sure that no new PRs have merged in the meantime, and no PRs are in + flight and soon to be merged. +1. Create a new release following a previous release as a template. Be sure to select the correct + branch. This requires Github release permissions as required by the prerequisites. + [external-provisioner example](https://github.com/kubernetes-csi/external-provisioner/releases/new) +1. If release was a new major/minor version, create a new `release-` + branch at that commit. +1. Update [kubernetes-csi/docs](https://github.com/kubernetes-csi/docs) sidecar + and feature pages with the new released version. +1. After all the sidecars have been released, update + CSI hostpath driver with the new sidecars in the [CSI repo](https://github.com/kubernetes-csi/csi-driver-host-path/tree/master/deploy) + and [k/k + in-tree](https://github.com/kubernetes/kubernetes/tree/master/test/e2e/testing-manifests/storage-csi/hostpath/hostpath) From fa90abd07862e6fa0a7b66ea2435b9db364ce6c9 Mon Sep 17 00:00:00 2001 From: wangzheng03 Date: Sun, 19 Jan 2020 09:42:26 +0800 Subject: [PATCH 080/107] fix incorrect link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 08c8288..60eab2a 100644 --- a/README.md +++ b/README.md @@ -81,7 +81,7 @@ on what is enabled in Prow, see https://github.com/kubernetes/test-infra/tree/master/config/jobs/kubernetes-csi Test results for periodic jobs are visible in -https://testgrid.k8s.io/sig-storage-csi +https://testgrid.k8s.io/sig-storage-csi-ci It is possible to reproduce the Prow testing locally on a suitable machine: - Linux host From 84f78b120e891b76b3e368cb0d20502baec026ea Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Mon, 10 Feb 2020 13:08:33 +0100 Subject: [PATCH 081/107] prow.sh: generic driver installation This relies on a slightly different deployment script: a "deploy.sh" must exist which knows that it has to dump a test driver configurion into the file pointed to with CSI_PROW_TEST_DRIVER, if that env variable is set. That way, we no longer need to know what capabilities the installed driver has. --- prow.sh | 104 +++++++++++++++++++++++--------------------------------- 1 file changed, 43 insertions(+), 61 deletions(-) diff --git a/prow.sh b/prow.sh index 118c5bd..bb80741 100755 --- a/prow.sh +++ b/prow.sh @@ -157,7 +157,9 @@ csi_prow_kubernetes_version_suffix="$(echo "${CSI_PROW_KUBERNETES_VERSION}" | tr # the caller. configvar CSI_PROW_WORK "$(mkdir -p "$GOPATH/pkg" && mktemp -d "$GOPATH/pkg/csiprow.XXXXXXXXXX")" "work directory" -# The hostpath deployment script is searched for in several places. +# By default, this script tests sidecars with the CSI hostpath driver, +# using the install_csi_driver function. That function depends on +# a deployment script that it searches for in several places: # # - The "deploy" directory in the current repository: this is useful # for the situation that a component becomes incompatible with the @@ -165,11 +167,11 @@ configvar CSI_PROW_WORK "$(mkdir -p "$GOPATH/pkg" && mktemp -d "$GOPATH/pkg/csip # own example until the shared one can be updated; it's also how # csi-driver-host-path itself provides the example. # -# - CSI_PROW_HOSTPATH_VERSION of the CSI_PROW_HOSTPATH_REPO is checked +# - CSI_PROW_DRIVER_VERSION of the CSI_PROW_DRIVER_REPO is checked # out: this allows other repos to reference a version of the example # that is known to be compatible. # -# - The csi-driver-host-path/deploy directory has multiple sub-directories, +# - The /deploy directory can have multiple sub-directories, # each with different deployments (stable set of images for Kubernetes 1.13, # stable set of images for Kubernetes 1.14, canary for latest Kubernetes, etc.). # This is necessary because there may be incompatible changes in the @@ -186,16 +188,26 @@ configvar CSI_PROW_WORK "$(mkdir -p "$GOPATH/pkg" && mktemp -d "$GOPATH/pkg/csip # "none" disables the deployment of the hostpath driver. # # When no deploy script is found (nothing in `deploy` directory, -# CSI_PROW_HOSTPATH_REPO=none), nothing gets deployed. -configvar CSI_PROW_HOSTPATH_VERSION "v1.3.0-rc3" "hostpath driver" -configvar CSI_PROW_HOSTPATH_REPO https://github.com/kubernetes-csi/csi-driver-host-path "hostpath repo" +# CSI_PROW_DRIVER_REPO=none), nothing gets deployed. +# +# If the deployment script is called with CSI_PROW_TEST_DRIVER= as +# environment variable, then it must write a suitable test driver configuration +# into that file in addition to installing the driver. +configvar CSI_PROW_DRIVER_VERSION "v1.3.0-rc4" "CSI driver version" +configvar CSI_PROW_DRIVER_REPO https://github.com/kubernetes-csi/csi-driver-host-path "CSI driver repo" configvar CSI_PROW_DEPLOYMENT "" "deployment" -configvar CSI_PROW_HOSTPATH_DRIVER_NAME "hostpath.csi.k8s.io" "the hostpath driver name" -# If CSI_PROW_HOSTPATH_CANARY is set (typically to "canary", but also -# "1.0-canary"), then all image versions are replaced with that -# version tag. -configvar CSI_PROW_HOSTPATH_CANARY "" "hostpath image" +# The install_csi_driver function may work also for other CSI drivers, +# as long as they follow the conventions of the CSI hostpath driver. +# If they don't, then a different install function can be provided in +# a .prow.sh file and this config variable can be overridden. +configvar CSI_PROW_DRIVER_INSTALL "install_csi_driver" "name of the shell function which installs the CSI driver" + +# If CSI_PROW_DRIVER_CANARY is set (typically to "canary", but also +# version tag. Usually empty. CSI_PROW_HOSTPATH_CANARY is +# accepted as alternative name because some test-infra jobs +# still use that name. +configvar CSI_PROW_DRIVER_CANARY "${CSI_PROW_HOSTPATH_CANARY}" "driver image override for canary images" # The E2E testing can come from an arbitrary repo. The expectation is that # the repo supports "go test ./test/e2e -args --storage.testdriver" (https://github.com/kubernetes/kubernetes/pull/72836) @@ -613,7 +625,7 @@ find_deployment () { # Fixed deployment name? Use it if it exists, otherwise fail. if [ "${CSI_PROW_DEPLOYMENT}" ]; then - file="$dir/${CSI_PROW_DEPLOYMENT}/deploy-hostpath.sh" + file="$dir/${CSI_PROW_DEPLOYMENT}/deploy.sh" if ! [ -e "$file" ]; then return 1 fi @@ -623,9 +635,9 @@ find_deployment () { # Ignore: See if you can use ${variable//search/replace} instead. # shellcheck disable=SC2001 - file="$dir/kubernetes-$(echo "${CSI_PROW_KUBERNETES_VERSION}" | sed -e 's/\([0-9]*\)\.\([0-9]*\).*/\1.\2/')/deploy-hostpath.sh" + file="$dir/kubernetes-$(echo "${CSI_PROW_KUBERNETES_VERSION}" | sed -e 's/\([0-9]*\)\.\([0-9]*\).*/\1.\2/')/deploy.sh" if ! [ -e "$file" ]; then - file="$dir/kubernetes-latest/deploy-hostpath.sh" + file="$dir/kubernetes-latest/deploy.sh" if ! [ -e "$file" ]; then return 1 fi @@ -633,12 +645,11 @@ find_deployment () { echo "$file" } -# This installs the hostpath driver example. CSI_PROW_HOSTPATH_CANARY overrides all -# image versions with that canary version. The parameters of install_hostpath can be -# used to override registry and/or tag of individual images (CSI_PROVISIONER_REGISTRY=localhost:9000 -# CSI_PROVISIONER_TAG=latest). -install_hostpath () { - local images deploy_hostpath +# This installs the CSI driver. It's called with a list of env variables +# that override the default images. CSI_PROW_DRIVER_CANARY overrides all +# image versions with that canary version. +install_csi_driver () { + local images deploy_driver images="$*" if [ "${CSI_PROW_DEPLOYMENT}" = "none" ]; then @@ -654,31 +665,31 @@ install_hostpath () { done fi - if deploy_hostpath="$(find_deployment "$(pwd)/deploy")"; then + if deploy_driver="$(find_deployment "$(pwd)/deploy")"; then : - elif [ "${CSI_PROW_HOSTPATH_REPO}" = "none" ]; then + elif [ "${CSI_PROW_DRIVER_REPO}" = "none" ]; then return 1 else - git_checkout "${CSI_PROW_HOSTPATH_REPO}" "${CSI_PROW_WORK}/hostpath" "${CSI_PROW_HOSTPATH_VERSION}" --depth=1 || die "checking out hostpath repo failed" - if deploy_hostpath="$(find_deployment "${CSI_PROW_WORK}/hostpath/deploy")"; then + git_checkout "${CSI_PROW_DRIVER_REPO}" "${CSI_PROW_WORK}/csi-driver" "${CSI_PROW_DRIVER_VERSION}" --depth=1 || die "checking out CSI driver repo failed" + if deploy_driver="$(find_deployment "${CSI_PROW_WORK}/csi-driver/deploy")"; then : else - die "deploy-hostpath.sh not found in ${CSI_PROW_HOSTPATH_REPO} ${CSI_PROW_HOSTPATH_VERSION}. To disable E2E testing, set CSI_PROW_HOSTPATH_REPO=none" + die "deploy.sh not found in ${CSI_PROW_DRIVER_REPO} ${CSI_PROW_DRIVER_VERSION}. To disable E2E testing, set CSI_PROW_DRIVER_REPO=none" fi fi - if [ "${CSI_PROW_HOSTPATH_CANARY}" != "stable" ]; then - images="$images IMAGE_TAG=${CSI_PROW_HOSTPATH_CANARY}" + if [ "${CSI_PROW_DRIVER_CANARY}" != "stable" ]; then + images="$images IMAGE_TAG=${CSI_PROW_DRIVER_CANARY}" fi # Ignore: Double quote to prevent globbing and word splitting. # It's intentional here for $images. # shellcheck disable=SC2086 - if ! run env $images "${deploy_hostpath}"; then + if ! run env "CSI_PROW_TEST_DRIVER=${CSI_PROW_WORK}/test-driver.yaml" $images "${deploy_driver}"; then # Collect information about failed deployment before failing. collect_cluster_info (start_loggers >/dev/null; wait) info "For container output see job artifacts." - die "deploying the hostpath driver with ${deploy_hostpath} failed" + die "deploying the CSI driver with ${deploy_driver} failed" fi } @@ -804,33 +815,6 @@ install_sanity () ( run_with_go "${CSI_PROW_GO_VERSION_SANITY}" go test -c -o "${CSI_PROW_WORK}/csi-sanity" "${CSI_PROW_SANITY_IMPORT_PATH}/cmd/csi-sanity" || die "building csi-sanity failed" ) -# The default implementation of this function generates a external -# driver test configuration for the hostpath driver. -# -# The content depends on both what the E2E suite expects and what the -# installed hostpath driver supports. Generating it here seems prone -# to breakage, but it is uncertain where a better place might be. -generate_test_driver () { - cat <"${CSI_PROW_WORK}/test-driver.yaml" || die "generating test-driver.yaml failed" - # Rename, merge and filter JUnit files. Necessary in case that we run the E2E suite again # and to avoid the large number of "skipped" tests that we get from using # the full Kubernetes E2E testsuite while only running a few tests. @@ -1063,7 +1045,7 @@ main () { cmds="$(grep '^\s*CMDS\s*=' Makefile | sed -e 's/\s*CMDS\s*=//')" # Get the image that was just built (if any) from the # top-level Makefile CMDS variable and set the - # deploy-hostpath.sh env variables for it. We also need to + # deploy.sh env variables for it. We also need to # side-load those images into the cluster. for i in $cmds; do e=$(echo "$i" | tr '[:lower:]' '[:upper:]' | tr - _) @@ -1101,7 +1083,7 @@ main () { fi # Installing the driver might be disabled. - if install_hostpath "$images"; then + if ${CSI_PROW_DRIVER_INSTALL} "$images"; then collect_cluster_info if sanity_enabled; then @@ -1158,7 +1140,7 @@ main () { fi # Installing the driver might be disabled. - if install_hostpath "$images"; then + if ${CSI_PROW_DRIVER_INSTALL} "$images"; then collect_cluster_info if tests_enabled "parallel-alpha"; then From 5f74333a466f0767cec812945c970c943a59cf7f Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Mon, 10 Feb 2020 11:06:31 +0100 Subject: [PATCH 082/107] prow.sh: also configure feature gates for kubelet That this hasn't been done before is an oversight. Apparently it hasn't been a problem because there never have been feature gates that mattered? --- prow.sh | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/prow.sh b/prow.sh index 118c5bd..6778e8b 100755 --- a/prow.sh +++ b/prow.sh @@ -568,6 +568,13 @@ kubeadmConfigPatches: nodeRegistration: kubeletExtraArgs: "feature-gates": "$gates" +- | + apiVersion: kubelet.config.k8s.io/v1beta1 + kind: KubeletConfiguration + metadata: + name: config + featureGates: +$(list_gates "$gates") - | apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration From fdb32183fea99d452a2e110ef9ca6bfd6f3e9fa3 Mon Sep 17 00:00:00 2001 From: Jan Wozniak Date: Thu, 13 Feb 2020 11:21:45 +0100 Subject: [PATCH 083/107] Change 'make test-vet' to call 'go vet' --- build.make | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.make b/build.make index a9b9d25..3bf3391 100644 --- a/build.make +++ b/build.make @@ -113,7 +113,7 @@ test-go: test: test-vet test-vet: @ echo; echo "### $@:" - go test $(GOFLAGS_VENDOR) `go list $(GOFLAGS_VENDOR) ./... | grep -v vendor $(TEST_VET_FILTER_CMD)` + go vet $(GOFLAGS_VENDOR) `go list $(GOFLAGS_VENDOR) ./... | grep -v vendor $(TEST_VET_FILTER_CMD)` .PHONY: test-fmt test: test-fmt From 7c5a89c8fcd634c4413837bc1042bd4bf27a93e6 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Fri, 14 Feb 2020 09:38:16 +0100 Subject: [PATCH 084/107] prow.sh: use 1.3.0 hostpath driver for testing The final 1.3.0 release of the hostpath driver really uses the 1.3.0 driver image in its deployment, in contrast to the previous -rc candidates which still used 1.2.0. --- prow.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prow.sh b/prow.sh index 0331ac3..d54dea5 100755 --- a/prow.sh +++ b/prow.sh @@ -193,7 +193,7 @@ configvar CSI_PROW_WORK "$(mkdir -p "$GOPATH/pkg" && mktemp -d "$GOPATH/pkg/csip # If the deployment script is called with CSI_PROW_TEST_DRIVER= as # environment variable, then it must write a suitable test driver configuration # into that file in addition to installing the driver. -configvar CSI_PROW_DRIVER_VERSION "v1.3.0-rc4" "CSI driver version" +configvar CSI_PROW_DRIVER_VERSION "v1.3.0" "CSI driver version" configvar CSI_PROW_DRIVER_REPO https://github.com/kubernetes-csi/csi-driver-host-path "CSI driver repo" configvar CSI_PROW_DEPLOYMENT "" "deployment" From 3863a0f674b70d87f34be95cad6dd464118713fa Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Wed, 4 Mar 2020 11:19:09 +0100 Subject: [PATCH 085/107] build for multiple platforms only in CI, add s390x Developers should not be forced to build for all platforms by default. We also don't want to copy-and-paste the go invocation for each new platform. To address both, the target platform(s) are now configurable via BUILD_PLATFORMS and additional platforms are only enabled in the Prow CI. For now this serves as a test that the source actually compiles for multiple platforms. Building images for different target platforms is a different problem. --- build.make | 21 ++++++++++++++------- prow.sh | 4 +++- 2 files changed, 17 insertions(+), 8 deletions(-) diff --git a/build.make b/build.make index 3bf3391..5c9793c 100644 --- a/build.make +++ b/build.make @@ -60,18 +60,25 @@ else TESTARGS = endif -ARCH := $(if $(GOARCH),$(GOARCH),$(shell go env GOARCH)) - # Specific packages can be excluded from each of the tests below by setting the *_FILTER_CMD variables # to something like "| grep -v 'github.com/kubernetes-csi/project/pkg/foobar'". See usage below. +# BUILD_PLATFORMS contains a set of triplets, +# separated by semicolon. An empty variable or empty entry (= just a +# semicolon) builds for the default platform of the current Go +# toolchain. +BUILD_PLATFORMS = + +# This builds each command (= the sub-directories of ./cmd) for the target platform(s) +# defined by BUILD_PLATFORMS. build-%: check-go-version-go mkdir -p bin - CGO_ENABLED=0 GOOS=linux go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$* ./cmd/$* - if [ "$$ARCH" = "amd64" ]; then \ - CGO_ENABLED=0 GOOS=windows go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$*.exe ./cmd/$* ; \ - CGO_ENABLED=0 GOOS=linux GOARCH=ppc64le go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$*-ppc64le ./cmd/$* ; \ - fi + echo '$(BUILD_PLATFORMS)' | tr ';' '\n' | while read -r os arch suffix; do \ + if ! (set -x; CGO_ENABLED=0 GOOS="$$os" GOARCH="$$arch" go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o "./bin/$*$$suffix" ./cmd/$*); then \ + echo "Building $* for GOOS=$$os GOARCH=$$arch failed, see error(s) above."; \ + exit 1; \ + fi; \ + done container-%: build-% docker build -t $*:latest -f $(shell if [ -e ./cmd/$*/Dockerfile ]; then echo ./cmd/$*/Dockerfile; else echo Dockerfile; fi) --label revision=$(REV) . diff --git a/prow.sh b/prow.sh index d54dea5..30cb384 100755 --- a/prow.sh +++ b/prow.sh @@ -85,6 +85,8 @@ get_versioned_variable () { echo "$value" } +configvar CSI_PROW_BUILD_PLATFORMS "linux amd64; windows amd64 .exe; linux ppc64le -ppc64le; linux s390x -s390x" "Go target platforms (= GOOS + GOARCH) and file suffix of the resulting binaries" + # If we have a vendor directory, then use it. We must be careful to only # use this for "make" invocations inside the project's repo itself because # setting it globally can break other go usages (like "go get " @@ -1026,7 +1028,7 @@ main () { images= if ${CSI_PROW_BUILD_JOB}; then # A successful build is required for testing. - run_with_go "${CSI_PROW_GO_VERSION_BUILD}" make all "GOFLAGS_VENDOR=${GOFLAGS_VENDOR}" || die "'make all' failed" + run_with_go "${CSI_PROW_GO_VERSION_BUILD}" make all "GOFLAGS_VENDOR=${GOFLAGS_VENDOR}" "BUILD_PLATFORMS=${CSI_PROW_BUILD_PLATFORMS}" || die "'make all' failed" # We don't want test failures to prevent E2E testing below, because the failure # might have been minor or unavoidable, for example when experimenting with # changes in "release-tools" in a PR (that fails the "is release-tools unmodified" From 7edc1461eacefed7f3112ac65ebefbdd2c4a64d1 Mon Sep 17 00:00:00 2001 From: Grant Griffiths Date: Wed, 11 Mar 2020 10:48:03 -0700 Subject: [PATCH 086/107] Update snapshotter to version 2.0.1 Signed-off-by: Grant Griffiths --- prow.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prow.sh b/prow.sh index 30cb384..86b2bf0 100755 --- a/prow.sh +++ b/prow.sh @@ -342,7 +342,7 @@ configvar CSI_PROW_E2E_ALPHA_GATES_LATEST '' "alpha feature gates for latest Kub configvar CSI_PROW_E2E_ALPHA_GATES "$(get_versioned_variable CSI_PROW_E2E_ALPHA_GATES "${csi_prow_kubernetes_version_suffix}")" "alpha E2E feature gates" # Which external-snapshotter tag to use for the snapshotter CRD and snapshot-controller deployment -configvar CSI_SNAPSHOTTER_VERSION 'v2.0.0' "external-snapshotter version tag" +configvar CSI_SNAPSHOTTER_VERSION 'v2.0.1' "external-snapshotter version tag" # Some tests are known to be unusable in a KinD cluster. For example, # stopping kubelet with "ssh systemctl stop kubelet" simply From ea1f94aad7b211ecd18f05794582c60869bf9251 Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Fri, 17 Apr 2020 09:02:07 -0700 Subject: [PATCH 087/107] update release tools instructions --- SIDECAR_RELEASE_PROCESS.md | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/SIDECAR_RELEASE_PROCESS.md b/SIDECAR_RELEASE_PROCESS.md index 91a1e1b..c8aaa4e 100644 --- a/SIDECAR_RELEASE_PROCESS.md +++ b/SIDECAR_RELEASE_PROCESS.md @@ -54,14 +54,21 @@ naming convention `-on-`. generator](https://github.com/kubernetes/release/tree/master/cmd/release-notes) 1. Generate release notes for the release. Replace arguments with the relevant information. - ``` - GITHUB_TOKEN= ./release-notes --start-sha=0ed6978fd199e3ca10326b82b4b8b8e916211c9b --end-sha=3cb3d2f18ed8cb40371c6d8886edcabd1f27e7b9 \ - --github-org=kubernetes-csi --github-repo=external-attacher -branch=master -output out.md - ``` - * `--start-sha` should point to the last release from the same branch. For - example: - * `1.X-1.0` tag when releasing `1.X.0` - * `1.X.Y-1` tag when releasing `1.X.Y` + * For new minor releases on master: + ``` + GITHUB_TOKEN= release-notes --discover=mergebase-to-latest + --github-org=kubernetes-csi --github-repo=external-provisioner + --required-author="" --output out.md + ``` + * For new patch releases on a release branch: + ``` + GITHUB_TOKEN= release-notes --branch=release-1.1 + --start-rev=v1.1.1 --end-sha=f0a9219b29cc9053047c39d149ce9b22bc7b918b + --github-org=kubernetes-csi --github-repo=external-provisioner + --required-author="" --output out.md + ``` + * `--start-rev` should point to the last patch release from the release branch. + * `--end-sha` should point to the latest commit from the release branch. 1. Compare the generated output to the new commits for the release to check if any notable change missed a release note. 1. Reword release notes as needed. Make sure to check notes for breaking From d8c76fee3e64e38a3427c9cf9df0f069801e3d74 Mon Sep 17 00:00:00 2001 From: Grant Griffiths Date: Wed, 15 Apr 2020 19:11:33 -0700 Subject: [PATCH 088/107] Support local snapshot RBAC for pull jobs Signed-off-by: Grant Griffiths --- prow.sh | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/prow.sh b/prow.sh index 86b2bf0..0b962f0 100755 --- a/prow.sh +++ b/prow.sh @@ -1064,18 +1064,24 @@ main () { # always pulling the image # (https://github.com/kubernetes-sigs/kind/issues/328). docker tag "$i:latest" "$i:csiprow" || die "tagging the locally built container image for $i failed" - done - if [ -e deploy/kubernetes/rbac.yaml ]; then - # This is one of those components which has its own RBAC rules (like external-provisioner). - # We are testing a locally built image and also want to test with the the current, - # potentially modified RBAC rules. - if [ "$(echo "$cmds" | wc -w)" != 1 ]; then - die "ambiguous deploy/kubernetes/rbac.yaml: need exactly one command, got: $cmds" + # For components with multiple cmds, the RBAC file should be in the following format: + # rbac-$cmd.yaml + # If this file cannot be found, we can default to the standard location: + # deploy/kubernetes/rbac.yaml + rbac_file_path=$(find . -type f -name "rbac-$i.yaml") + if [ "$rbac_file_path" == "" ]; then + rbac_file_path="$(pwd)/deploy/kubernetes/rbac.yaml" fi - e=$(echo "$cmds" | tr '[:lower:]' '[:upper:]' | tr - _) - images="$images ${e}_RBAC=$(pwd)/deploy/kubernetes/rbac.yaml" - fi + + if [ -e "$rbac_file_path" ]; then + # This is one of those components which has its own RBAC rules (like external-provisioner). + # We are testing a locally built image and also want to test with the the current, + # potentially modified RBAC rules. + e=$(echo "$i" | tr '[:lower:]' '[:upper:]' | tr - _) + images="$images ${e}_RBAC=$rbac_file_path" + fi + done fi if tests_need_non_alpha_cluster; then From 6f2322e80813459de15eb2aa0d17ddd53cb3efa8 Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Thu, 23 Apr 2020 10:21:49 -0700 Subject: [PATCH 089/107] Update patch release notes generation command --- SIDECAR_RELEASE_PROCESS.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/SIDECAR_RELEASE_PROCESS.md b/SIDECAR_RELEASE_PROCESS.md index c8aaa4e..6960607 100644 --- a/SIDECAR_RELEASE_PROCESS.md +++ b/SIDECAR_RELEASE_PROCESS.md @@ -50,7 +50,7 @@ naming convention `-on-`. ## Release Process 1. Identify all issues and ongoing PRs that should go into the release, and drive them to resolution. -1. Download [K8s release notes +1. Download v2.8+ [K8s release notes generator](https://github.com/kubernetes/release/tree/master/cmd/release-notes) 1. Generate release notes for the release. Replace arguments with the relevant information. @@ -62,13 +62,10 @@ naming convention `-on-`. ``` * For new patch releases on a release branch: ``` - GITHUB_TOKEN= release-notes --branch=release-1.1 - --start-rev=v1.1.1 --end-sha=f0a9219b29cc9053047c39d149ce9b22bc7b918b + GITHUB_TOKEN= release-notes --discover=patch-to-latest --branch=release-1.1 --github-org=kubernetes-csi --github-repo=external-provisioner --required-author="" --output out.md ``` - * `--start-rev` should point to the last patch release from the release branch. - * `--end-sha` should point to the latest commit from the release branch. 1. Compare the generated output to the new commits for the release to check if any notable change missed a release note. 1. Reword release notes as needed. Make sure to check notes for breaking From bd416901d4b7cad7bc70a39f508b68e2a9fff1f6 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Thu, 28 May 2020 10:22:20 +0200 Subject: [PATCH 090/107] cloud build: initial set of shared files The approach taken here extends the existing support for cross-compiling binaries on the build host and specifying the Go compiler: Go is installed if needed (as in Prow testing), binaries are build on the host, then one image is created for each platform, and finally those are combined into a single multi-architecture image. --- build.make | 63 +++++++++++++++++++++++++++++++++++++++++++++++++ cloudbuild.sh | 6 +++++ cloudbuild.yaml | 44 ++++++++++++++++++++++++++++++++++ prow.sh | 14 +++++++++++ 4 files changed, 127 insertions(+) create mode 100755 cloudbuild.sh create mode 100644 cloudbuild.yaml diff --git a/build.make b/build.make index 5c9793c..e3a44f9 100644 --- a/build.make +++ b/build.make @@ -105,6 +105,69 @@ build: $(CMDS:%=build-%) container: $(CMDS:%=container-%) push: $(CMDS:%=push-%) +# Additional parameters are needed when pushing to a local registry, +# see https://github.com/docker/buildx/issues/94. +# However, that then runs into https://github.com/docker/cli/issues/2396. +# +# What works for local testing is: +# make push-multiarch PULL_BASE_REF=master REGISTRY_NAME= BUILD_PLATFORMS="linux amd64; windows amd64 .exe; linux ppc64le -ppc64le; linux s390x -s390x" +DOCKER_BUILDX_CREATE_ARGS ?= + +# This target builds a multiarch image for one command using Moby BuildKit builder toolkit. +# Docker Buildx is included in Docker 19.03. +# +# ./cmd//Dockerfile[.Windows] is used if found, otherwise Dockerfile[.Windows]. +# BUILD_PLATFORMS determines which individual images are included in the multiarch image. +# PULL_BASE_REF must be set to 'master', 'release-x.y', or a tag name, and determines +# the tag for the resulting multiarch image. +push-multiarch-%: check-pull-base-ref build-% + set -ex; \ + DOCKER_CLI_EXPERIMENTAL=enabled; \ + export DOCKER_CLI_EXPERIMENTAL; \ + docker buildx create $(DOCKER_BUILDX_CREATE_ARGS) --use --name multiarchimage-buildertest; \ + trap "docker buildx rm multiarchimage-buildertest" EXIT; \ + dockerfile_linux=$$(if [ -e ./cmd/$*/Dockerfile ]; then echo ./cmd/$*/Dockerfile; else echo Dockerfile; fi); \ + dockerfile_windows=$$(if [ -e ./cmd/$*/Dockerfile.Windows ]; then echo ./cmd/$*/Dockerfile.Windows; else echo Dockerfile.Windows; fi); \ + if [ '$(BUILD_PLATFORMS)' ]; then build_platforms='$(BUILD_PLATFORMS)'; else build_platforms="linux amd64"; fi; \ + pushMultiArch () { \ + tag=$$1; \ + echo "$$build_platforms" | tr ';' '\n' | while read -r os arch suffix; do \ + docker buildx build --push \ + --tag $(IMAGE_NAME):$$arch-$$os-$$tag \ + --platform=$$os/$$arch \ + --file $$(eval echo \$${dockerfile_$$os}) \ + --build-arg binary=./bin/$*$$suffix \ + --label revision=$(REV) \ + .; \ + done; \ + images=$$(echo "$$build_platforms" | tr ';' '\n' | while read -r os arch suffix; do echo $(IMAGE_NAME):$$arch-$$os-$$tag; done); \ + docker manifest create --amend $(IMAGE_NAME):$$tag $$images; \ + docker manifest push -p $(IMAGE_NAME):$$tag; \ + }; \ + if [ $(PULL_BASE_REF) = "master" ]; then \ + : "creating or overwriting canary image"; \ + pushMultiArch canary; \ + elif echo $(PULL_BASE_REF) | grep -q -e 'release-*' ; then \ + : "creating or overwriting canary image for release branch"; \ + release_canary_tag=$$(echo $(PULL_BASE_REF) | cut -f2 -d '-')-canary; \ + pushMultiArch $$release_canary_tag; \ + elif docker pull $(IMAGE_NAME):$(PULL_BASE_REF) 2>&1 | tee /dev/stderr | grep -q "manifest for $(IMAGE_NAME):$(PULL_BASE_REF) not found"; then \ + : "creating release image"; \ + pushMultiArch $(PULL_BASE_REF); \ + else \ + : "ERROR: release image $(IMAGE_NAME):$(PULL_BASE_REF) already exists: a new tag is required!"; \ + exit 1; \ + fi + +.PHONY: check-pull-base-ref +check-pull-base-ref: + if ! [ "$(PULL_BASE_REF)" ]; then \ + echo >&2 "ERROR: PULL_BASE_REF must be set to 'master', 'release-x.y', or a tag name."; \ + exit 1; \ + fi + +push-multiarch: $(CMDS:%=push-multiarch-%) + clean: -rm -rf bin diff --git a/cloudbuild.sh b/cloudbuild.sh new file mode 100755 index 0000000..3ba11ec --- /dev/null +++ b/cloudbuild.sh @@ -0,0 +1,6 @@ +#! /bin/bash + +# shellcheck disable=SC1091 +. release-tools/prow.sh + +gcr_cloud_build diff --git a/cloudbuild.yaml b/cloudbuild.yaml new file mode 100644 index 0000000..1def149 --- /dev/null +++ b/cloudbuild.yaml @@ -0,0 +1,44 @@ +# A configuration file for multi-arch image building with the Google cloud build service. +# +# Repos using this file must: +# - import csi-release-tools +# - add a symlink cloudbuild.yaml -> release-tools/cloudbuild.yaml +# - add a .cloudbuild.sh which can be a custom file or a symlink +# to release-tools/cloudbuild.sh +# - accept "binary" as build argument in their Dockerfile(s) (see +# https://github.com/pohly/node-driver-registrar/blob/3018101987b0bb6da2a2657de607174d6e3728f7/Dockerfile#L4-L6) +# because binaries will get built for different architectures and then +# get copied from the built host into the container image +# +# See https://github.com/kubernetes/test-infra/blob/master/config/jobs/image-pushing/README.md +# for more details on image pushing process in Kubernetes. + +# This must be specified in seconds. If omitted, defaults to 600s (10 mins). +timeout: 1200s +# This prevents errors if you don't use both _GIT_TAG and _PULL_BASE_REF, +# or any new substitutions added in the future. +options: + substitution_option: ALLOW_LOOSE +steps: + # The image must contain bash and curl. Ideally it should also contain + # the desired version of Go (currently defined in release-tools/travis.yml), + # but that just speeds up the build and is not required. + - name: 'gcr.io/k8s-testimages/gcb-docker-gcloud:v20200421-a2bf5f8' + entrypoint: ./.cloudbuild.sh + env: + - GIT_TAG=${_GIT_TAG} + - PULL_BASE_REF=${_PULL_BASE_REF} + - REGISTRY_NAME=gcr.io/${_STAGING_PROJECT} + - HOME=/root +substitutions: + # _GIT_TAG will be filled with a git-based tag for the image, of the form vYYYYMMDD-hash, and + # can be used as a substitution. + _GIT_TAG: '12345' + # _PULL_BASE_REF will contain the ref that was pushed to trigger this build - + # a branch like 'master' or 'release-0.2', or a tag like 'v0.2'. + _PULL_BASE_REF: 'master' + # The default gcr.io staging project for Kubernetes-CSI + # (=> https://console.cloud.google.com/gcr/images/k8s-staging-csi/GLOBAL). + # Might be overridden in the Prow build job for a repo which wants + # images elsewhere. + _STAGING_PROJECT: 'k8s-staging-csi' diff --git a/prow.sh b/prow.sh index 0b962f0..93964e1 100755 --- a/prow.sh +++ b/prow.sh @@ -1189,3 +1189,17 @@ main () { return "$ret" } + +# This function can be called by a repo's top-level cloudbuild.sh: +# it handles environment set up in the GCR cloud build and then +# invokes "make push-multiarch" to do the actual image building. +gcr_cloud_build () { + # Register gcloud as a Docker credential helper. + # Required for "docker buildx build --push". + gcloud auth configure-docker + + # Extract tag-n-hash value from GIT_TAG (form vYYYYMMDD-tag-n-hash) for REV value. + REV=v$(echo "$GIT_TAG" | cut -f3- -d 'v') + + run_with_go "${CSI_PROW_GO_VERSION_BUILD}" make push-multiarch REV="${REV}" REGISTRY_NAME="${REGISTRY_NAME}" BUILD_PLATFORMS="${CSI_PROW_BUILD_PLATFORMS}" +} From 4569f27a8cf57336ea13d8ef3dc4423c76712913 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Wed, 3 Jun 2020 11:03:47 +0200 Subject: [PATCH 091/107] build.make: fix push-multiarch ambiguity "make push-multiarch" matched both push-multiarch and push-%. This seems to be none-deterministic and in at least one repo (external-provisioner), make picked the wildcard rule which then failed because there is no "multiarch" command. This ambiguity gets resolved by instantiating the wildcard rules only for existing commands. The advantage also is that "make push-no-such-command" will fail with an obvious "No rule to make target 'push-no-such-command'" instead of attempting to build the command. --- build.make | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/build.make b/build.make index e3a44f9..0e4a9ba 100644 --- a/build.make +++ b/build.make @@ -71,7 +71,7 @@ BUILD_PLATFORMS = # This builds each command (= the sub-directories of ./cmd) for the target platform(s) # defined by BUILD_PLATFORMS. -build-%: check-go-version-go +$(CMDS:%=build-%): build-%: check-go-version-go mkdir -p bin echo '$(BUILD_PLATFORMS)' | tr ';' '\n' | while read -r os arch suffix; do \ if ! (set -x; CGO_ENABLED=0 GOOS="$$os" GOARCH="$$arch" go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o "./bin/$*$$suffix" ./cmd/$*); then \ @@ -80,10 +80,10 @@ build-%: check-go-version-go fi; \ done -container-%: build-% +$(CMDS:%=container-%): container-%: build-% docker build -t $*:latest -f $(shell if [ -e ./cmd/$*/Dockerfile ]; then echo ./cmd/$*/Dockerfile; else echo Dockerfile; fi) --label revision=$(REV) . -push-%: container-% +$(CMDS:%=push-%): push-%: container-% set -ex; \ push_image () { \ docker tag $*:latest $(IMAGE_NAME):$$tag; \ @@ -120,7 +120,7 @@ DOCKER_BUILDX_CREATE_ARGS ?= # BUILD_PLATFORMS determines which individual images are included in the multiarch image. # PULL_BASE_REF must be set to 'master', 'release-x.y', or a tag name, and determines # the tag for the resulting multiarch image. -push-multiarch-%: check-pull-base-ref build-% +$(CMDS:%=push-multiarch-%): push-multiarch-%: check-pull-base-ref build-% set -ex; \ DOCKER_CLI_EXPERIMENTAL=enabled; \ export DOCKER_CLI_EXPERIMENTAL; \ From 5231f05d8b2f7a6b656fb27d84abcb758000842b Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Wed, 3 Jun 2020 11:04:29 +0200 Subject: [PATCH 092/107] build.make: properly declare push-multiarch It's not a real file and thus should better be marked as phony. --- build.make | 1 + 1 file changed, 1 insertion(+) diff --git a/build.make b/build.make index 0e4a9ba..83d903e 100644 --- a/build.make +++ b/build.make @@ -166,6 +166,7 @@ check-pull-base-ref: exit 1; \ fi +.PHONY: push-multiarch push-multiarch: $(CMDS:%=push-multiarch-%) clean: From 340e082f04321801f51cdff42576bd55dcfc3683 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Wed, 3 Jun 2020 11:05:05 +0200 Subject: [PATCH 093/107] build.make: optional inclusion of Windows in multiarch images Most repos inherit the default BUILD_PLATFORMS, which includes Windows, but don't have the necessary Dockerfile.Windows yet. To simplify the rollout of multiarch image builds, Windows binary building continues to be tested (i.e. BUILD_PLATFORMS remains unchanged), but push-multiarch skips Windows if the Dockerfile.Windows is missing. --- build.make | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/build.make b/build.make index 83d903e..c17a1c1 100644 --- a/build.make +++ b/build.make @@ -117,6 +117,10 @@ DOCKER_BUILDX_CREATE_ARGS ?= # Docker Buildx is included in Docker 19.03. # # ./cmd//Dockerfile[.Windows] is used if found, otherwise Dockerfile[.Windows]. +# It is currently optional: if no such file exists, Windows images are not included, +# even when Windows is listed in BUILD_PLATFORMS. That way, projects can test that +# Windows binaries can be built before adding a Dockerfile for it. +# # BUILD_PLATFORMS determines which individual images are included in the multiarch image. # PULL_BASE_REF must be set to 'master', 'release-x.y', or a tag name, and determines # the tag for the resulting multiarch image. @@ -129,6 +133,9 @@ $(CMDS:%=push-multiarch-%): push-multiarch-%: check-pull-base-ref build-% dockerfile_linux=$$(if [ -e ./cmd/$*/Dockerfile ]; then echo ./cmd/$*/Dockerfile; else echo Dockerfile; fi); \ dockerfile_windows=$$(if [ -e ./cmd/$*/Dockerfile.Windows ]; then echo ./cmd/$*/Dockerfile.Windows; else echo Dockerfile.Windows; fi); \ if [ '$(BUILD_PLATFORMS)' ]; then build_platforms='$(BUILD_PLATFORMS)'; else build_platforms="linux amd64"; fi; \ + if ! [ -f "$$dockerfile_windows" ]; then \ + build_platforms="$$(echo "$$build_platforms" | sed -e 's/windows *[^ ]* *.exe//g' -e 's/; *;/;/g')"; \ + fi; \ pushMultiArch () { \ tag=$$1; \ echo "$$build_platforms" | tr ';' '\n' | while read -r os arch suffix; do \ From db0c2a7dc80d5b6d65ac8e4e8c64026f342fd987 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Thu, 4 Jun 2020 10:52:04 +0200 Subject: [PATCH 094/107] cloud build: initialize support for running commands in Dockerfile If the Dockerfile needs to run some command, that step fails unless QEMU is set up properly first: failed to solve: rpc error: code = Unknown desc = failed to load LLB: runtime execution on platform linux/ppc64le not supported --- prow.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/prow.sh b/prow.sh index 93964e1..32ec871 100755 --- a/prow.sh +++ b/prow.sh @@ -1198,6 +1198,12 @@ gcr_cloud_build () { # Required for "docker buildx build --push". gcloud auth configure-docker + if find . -name Dockerfile | grep -v ^./vendor | xargs --no-run-if-empty cat | grep -q ^RUN; then + # Needed for "RUN" steps on non-linux/amd64 platforms. + # See https://github.com/multiarch/qemu-user-static#getting-started + (set -x; docker run --rm --privileged multiarch/qemu-user-static --reset -p yes) + fi + # Extract tag-n-hash value from GIT_TAG (form vYYYYMMDD-tag-n-hash) for REV value. REV=v$(echo "$GIT_TAG" | cut -f3- -d 'v') From 3df86b7d437d7126a1ffb79ff05b0e467748b920 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Fri, 12 Jun 2020 15:48:26 +0200 Subject: [PATCH 095/107] cloud build: k8s-staging-sig-storage As discussed in https://github.com/kubernetes/k8s.io/pull/943, we want to consolidate under k8s-staging-sig-storage. --- cloudbuild.yaml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/cloudbuild.yaml b/cloudbuild.yaml index 1def149..a22c0e7 100644 --- a/cloudbuild.yaml +++ b/cloudbuild.yaml @@ -12,6 +12,8 @@ # # See https://github.com/kubernetes/test-infra/blob/master/config/jobs/image-pushing/README.md # for more details on image pushing process in Kubernetes. +# +# To promote release images, see https://github.com/kubernetes/k8s.io/tree/master/k8s.gcr.io/images/k8s-staging-sig-storage. # This must be specified in seconds. If omitted, defaults to 600s (10 mins). timeout: 1200s @@ -38,7 +40,7 @@ substitutions: # a branch like 'master' or 'release-0.2', or a tag like 'v0.2'. _PULL_BASE_REF: 'master' # The default gcr.io staging project for Kubernetes-CSI - # (=> https://console.cloud.google.com/gcr/images/k8s-staging-csi/GLOBAL). + # (=> https://console.cloud.google.com/gcr/images/k8s-staging-sig-storage/GLOBAL). # Might be overridden in the Prow build job for a repo which wants # images elsewhere. - _STAGING_PROJECT: 'k8s-staging-csi' + _STAGING_PROJECT: 'k8s-staging-sig-storage' From 43e50d6f6471f476acd5c90cd3788e198a86d9f8 Mon Sep 17 00:00:00 2001 From: Yibo Cai Date: Tue, 16 Jun 2020 10:45:09 +0800 Subject: [PATCH 096/107] prow.sh: enable building arm64 image --- prow.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prow.sh b/prow.sh index 32ec871..9778635 100755 --- a/prow.sh +++ b/prow.sh @@ -85,7 +85,7 @@ get_versioned_variable () { echo "$value" } -configvar CSI_PROW_BUILD_PLATFORMS "linux amd64; windows amd64 .exe; linux ppc64le -ppc64le; linux s390x -s390x" "Go target platforms (= GOOS + GOARCH) and file suffix of the resulting binaries" +configvar CSI_PROW_BUILD_PLATFORMS "linux amd64; windows amd64 .exe; linux ppc64le -ppc64le; linux s390x -s390x; linux arm64 -arm64" "Go target platforms (= GOOS + GOARCH) and file suffix of the resulting binaries" # If we have a vendor directory, then use it. We must be careful to only # use this for "make" invocations inside the project's repo itself because From 36ea4ffae0fabb9404ad1588be285fe739b6e565 Mon Sep 17 00:00:00 2001 From: Animesh Kumar Date: Wed, 17 Jun 2020 18:05:08 +0530 Subject: [PATCH 097/107] filter-junit: Fix golint error Signed-off-by: Animesh Kumar --- filter-junit.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/filter-junit.go b/filter-junit.go index 2f51be0..1da224a 100644 --- a/filter-junit.go +++ b/filter-junit.go @@ -15,9 +15,9 @@ limitations under the License. */ /* - * This command filters a JUnit file such that only tests with a name - * matching a regular expression are passed through. By concatenating - * multiple input files it is possible to merge them into a single file. +This command filters a JUnit file such that only tests with a name +matching a regular expression are passed through. By concatenating +multiple input files it is possible to merge them into a single file. */ package main From 5a54e67d80858259793f0b970c227153db1f4c06 Mon Sep 17 00:00:00 2001 From: Animesh Kumar Date: Thu, 18 Jun 2020 13:07:40 +0530 Subject: [PATCH 098/107] filter-junit: Fix gofmt error Signed-off-by: Animesh Kumar --- filter-junit.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/filter-junit.go b/filter-junit.go index 1da224a..cf1cb4a 100644 --- a/filter-junit.go +++ b/filter-junit.go @@ -18,7 +18,7 @@ limitations under the License. This command filters a JUnit file such that only tests with a name matching a regular expression are passed through. By concatenating multiple input files it is possible to merge them into a single file. - */ +*/ package main import ( From 1fdf2d53cddecb5e847ad810d0097e1e44a7b5f6 Mon Sep 17 00:00:00 2001 From: namrata-ibm Date: Wed, 8 Jul 2020 08:03:24 -0400 Subject: [PATCH 099/107] cloud build: bump timeout in Prow job --- cloudbuild.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cloudbuild.yaml b/cloudbuild.yaml index a22c0e7..8f67892 100644 --- a/cloudbuild.yaml +++ b/cloudbuild.yaml @@ -16,7 +16,7 @@ # To promote release images, see https://github.com/kubernetes/k8s.io/tree/master/k8s.gcr.io/images/k8s-staging-sig-storage. # This must be specified in seconds. If omitted, defaults to 600s (10 mins). -timeout: 1200s +timeout: 1800s # This prevents errors if you don't use both _GIT_TAG and _PULL_BASE_REF, # or any new substitutions added in the future. options: From 843bddca1493740fe733d44f749e8ba1e450ad72 Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Wed, 5 Aug 2020 13:29:03 -0700 Subject: [PATCH 100/107] Add steps on promoting release images --- SIDECAR_RELEASE_PROCESS.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/SIDECAR_RELEASE_PROCESS.md b/SIDECAR_RELEASE_PROCESS.md index 6960607..f6434c8 100644 --- a/SIDECAR_RELEASE_PROCESS.md +++ b/SIDECAR_RELEASE_PROCESS.md @@ -86,6 +86,12 @@ naming convention `-on-`. [external-provisioner example](https://github.com/kubernetes-csi/external-provisioner/releases/new) 1. If release was a new major/minor version, create a new `release-` branch at that commit. +1. Check [image build status](https://k8s-testgrid.appspot.com/sig-storage-image-build). +1. Promote images from k8s-staging-sig-storage to k8s.gcr.io/sig-storage. From + the [k8s image + repo](https://github.com/kubernetes/k8s.io/tree/master/k8s.gcr.io/images/k8s-staging-sig-storage), + run `./generate.sh > images.yaml`, and send a PR with the updated images. + Once merged, the image promoter will copy the images from staging to prod. 1. Update [kubernetes-csi/docs](https://github.com/kubernetes-csi/docs) sidecar and feature pages with the new released version. 1. After all the sidecars have been released, update From 82d108acdc4dd4cbd6d8d97534000dd5d8ea7e8d Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Wed, 12 Aug 2020 09:42:55 +0200 Subject: [PATCH 101/107] switch to Go 1.15 Go 1.15 was released and is the major version that Kubernetes 1.19.0 is going to use. There are probably bugs in the older 1.13.3 that were fixed, so we should update. --- travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/travis.yml b/travis.yml index bfd7647..1ab13ae 100644 --- a/travis.yml +++ b/travis.yml @@ -6,7 +6,7 @@ git: depth: false matrix: include: - - go: 1.13.3 + - go: 1.15 before_script: - mkdir -p bin - wget https://github.com/golang/dep/releases/download/v0.5.1/dep-linux-amd64 -O bin/dep From 3b4a2f1d956845ab094e61d8701eab2080226cd3 Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Fri, 14 Aug 2020 14:21:51 +0200 Subject: [PATCH 102/107] prow.sh: fix installing Go for Kubernetes 1.19.0 Kubernetes 1.19.0 uses Go 1.15, but refers to it as 1.15.0. This broke both the check whether we need to install 1.15 (because "go version" reports 1.15, which didn't match 1.15.0) and then downloading the release archive (because the URL also only uses 1.15). --- prow.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/prow.sh b/prow.sh index 9778635..8ce80cf 100755 --- a/prow.sh +++ b/prow.sh @@ -513,6 +513,10 @@ go_version_for_kubernetes () ( if ! [ "$go_version" ]; then die "Unable to determine Go version for Kubernetes $version from hack/lib/golang.sh." fi + # Strip the trailing .0. Kubernetes includes it, Go itself doesn't. + # Ignore: See if you can use ${variable//search/replace} instead. + # shellcheck disable=SC2001 + go_version="$(echo "$go_version" | sed -e 's/\.0$//')" echo "$go_version" ) From 0979c091026916e3593022951c08d4f79bdb2bdf Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Fri, 14 Aug 2020 14:23:51 +0200 Subject: [PATCH 103/107] prow.sh: fix E2E suite for Kubernetes >= 1.18 It used to be necessary to override from where the E2E suite came on a case-by-case basis (initially, testing was using a more recent suite against an older Kubernetes). This should never become necessary again and the lack of a specific entry for 1.18 already had the unintended effect that Kubernetes 1.18 was tested with the suite from master, so overall it is better to always use the E2E suite which matches Kubernetes. --- prow.sh | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/prow.sh b/prow.sh index 8ce80cf..05d397a 100755 --- a/prow.sh +++ b/prow.sh @@ -218,17 +218,18 @@ configvar CSI_PROW_DRIVER_CANARY "${CSI_PROW_HOSTPATH_CANARY}" "driver image ove # all generated files are present. # # CSI_PROW_E2E_REPO=none disables E2E testing. -# TOOO: remove versioned variables and make e2e version match k8s version -configvar CSI_PROW_E2E_VERSION_1_15 v1.15.0 "E2E version for Kubernetes 1.15.x" -configvar CSI_PROW_E2E_VERSION_1_16 v1.16.0 "E2E version for Kubernetes 1.16.x" -configvar CSI_PROW_E2E_VERSION_1_17 v1.17.0 "E2E version for Kubernetes 1.17.x" -# TODO: add new CSI_PROW_E2E_VERSION entry for future Kubernetes releases -configvar CSI_PROW_E2E_VERSION_LATEST master "E2E version for Kubernetes master" # testing against Kubernetes master is already tracking a moving target, so we might as well use a moving E2E version -configvar CSI_PROW_E2E_REPO_LATEST https://github.com/kubernetes/kubernetes "E2E repo for Kubernetes >= 1.13.x" # currently the same for all versions -configvar CSI_PROW_E2E_IMPORT_PATH_LATEST k8s.io/kubernetes "E2E package for Kubernetes >= 1.13.x" # currently the same for all versions -configvar CSI_PROW_E2E_VERSION "$(get_versioned_variable CSI_PROW_E2E_VERSION "${csi_prow_kubernetes_version_suffix}")" "E2E version" -configvar CSI_PROW_E2E_REPO "$(get_versioned_variable CSI_PROW_E2E_REPO "${csi_prow_kubernetes_version_suffix}")" "E2E repo" -configvar CSI_PROW_E2E_IMPORT_PATH "$(get_versioned_variable CSI_PROW_E2E_IMPORT_PATH "${csi_prow_kubernetes_version_suffix}")" "E2E package" +tag_from_version () { + version="$1" + shift + case "$version" in + latest) echo "master";; + release-*) echo "$version";; + *) echo "v$version";; + esac +} +configvar CSI_PROW_E2E_VERSION "$(tag_from_version "${CSI_PROW_KUBERNETES_VERSION}")" "E2E version" +configvar CSI_PROW_E2E_REPO "https://github.com/kubernetes/kubernetes" "E2E repo" +configvar CSI_PROW_E2E_IMPORT_PATH "k8s.io/kubernetes" "E2E package" # csi-sanity testing from the csi-test repo can be run against the installed # CSI driver. For this to work, deploying the driver must expose the Unix domain From 2c098465dd81ade3fbab1dd251f65d2ec4abc1a5 Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Mon, 24 Aug 2020 16:44:47 -0700 Subject: [PATCH 104/107] Add cleanup instructions to release-notes generation --- SIDECAR_RELEASE_PROCESS.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/SIDECAR_RELEASE_PROCESS.md b/SIDECAR_RELEASE_PROCESS.md index f6434c8..91e837c 100644 --- a/SIDECAR_RELEASE_PROCESS.md +++ b/SIDECAR_RELEASE_PROCESS.md @@ -54,14 +54,19 @@ naming convention `-on-`. generator](https://github.com/kubernetes/release/tree/master/cmd/release-notes) 1. Generate release notes for the release. Replace arguments with the relevant information. + * Clean up old cached information (also needed if you are generating release + notes for multiple repos) + ```bash + rm -rf /tmp/k8s-repo + ``` * For new minor releases on master: - ``` + ```bash GITHUB_TOKEN= release-notes --discover=mergebase-to-latest --github-org=kubernetes-csi --github-repo=external-provisioner --required-author="" --output out.md ``` * For new patch releases on a release branch: - ``` + ```bash GITHUB_TOKEN= release-notes --discover=patch-to-latest --branch=release-1.1 --github-org=kubernetes-csi --github-repo=external-provisioner --required-author="" --output out.md From e73c2ce5312662816b180c5b03eb490322051fd7 Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Fri, 25 Sep 2020 17:41:33 -0700 Subject: [PATCH 105/107] Use staging registry for canary tests --- prow.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/prow.sh b/prow.sh index 05d397a..1812e3a 100755 --- a/prow.sh +++ b/prow.sh @@ -211,6 +211,10 @@ configvar CSI_PROW_DRIVER_INSTALL "install_csi_driver" "name of the shell functi # still use that name. configvar CSI_PROW_DRIVER_CANARY "${CSI_PROW_HOSTPATH_CANARY}" "driver image override for canary images" +# Image registry to use for canary images. +# Only valid if CSI_PROW_DRIVER_CANARY is set. +configvar CSI_PROW_DRIVER_CANARY_REGISTRY "gcr.io/k8s-staging-sig-storage" "registry for canary images" + # The E2E testing can come from an arbitrary repo. The expectation is that # the repo supports "go test ./test/e2e -args --storage.testdriver" (https://github.com/kubernetes/kubernetes/pull/72836) # after setting KUBECONFIG. As a special case, if the repository is Kubernetes, @@ -693,7 +697,7 @@ install_csi_driver () { fi if [ "${CSI_PROW_DRIVER_CANARY}" != "stable" ]; then - images="$images IMAGE_TAG=${CSI_PROW_DRIVER_CANARY}" + images="$images IMAGE_TAG=${CSI_PROW_DRIVER_CANARY} IMAGE_REGISTRY=${CSI_PROW_DRIVER_CANARY_REGISTRY}" fi # Ignore: Double quote to prevent globbing and word splitting. # It's intentional here for $images. From d1294628784b0b37b9ce30aceab3725cd32c153e Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Fri, 25 Sep 2020 17:49:38 -0700 Subject: [PATCH 106/107] Document new method for adding CI jobs are new K8s versions --- SIDECAR_RELEASE_PROCESS.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/SIDECAR_RELEASE_PROCESS.md b/SIDECAR_RELEASE_PROCESS.md index 91e837c..4575eb8 100644 --- a/SIDECAR_RELEASE_PROCESS.md +++ b/SIDECAR_RELEASE_PROCESS.md @@ -39,10 +39,11 @@ naming convention `-on-`. 1. Changes can then be updated in all the sidecar repos and hostpath driver repo by following the [update instructions](https://github.com/kubernetes-csi/csi-release-tools/blob/master/README.md#sharing-and-updating). -1. New pull and CI jobs are configured by +1. New pull and CI jobs are configured by adding new K8s versions to the top of [gen-jobs.sh](https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-csi/gen-jobs.sh). - New pull jobs that have been unverified should be initially made optional. - [Example](https://github.com/kubernetes/test-infra/pull/15055) + New pull jobs that have been unverified should be initially made optional by + setting the new K8s version as + [experimental](https://github.com/kubernetes/test-infra/blob/a1858f46d6014480b130789df58b230a49203a64/config/jobs/kubernetes-csi/gen-jobs.sh#L40). 1. Once new pull and CI jobs have been verified, and the new Kubernetes version is released, we can make the optional jobs required, and also remove the Kubernetes versions that are no longer supported. From 7100c1209eba08b3444ad7cb84f31e225e0e3889 Mon Sep 17 00:00:00 2001 From: Michelle Au Date: Tue, 6 Oct 2020 11:25:19 -0700 Subject: [PATCH 107/107] Only set staging registry when running canary job --- prow.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/prow.sh b/prow.sh index 1812e3a..b069a31 100755 --- a/prow.sh +++ b/prow.sh @@ -212,7 +212,7 @@ configvar CSI_PROW_DRIVER_INSTALL "install_csi_driver" "name of the shell functi configvar CSI_PROW_DRIVER_CANARY "${CSI_PROW_HOSTPATH_CANARY}" "driver image override for canary images" # Image registry to use for canary images. -# Only valid if CSI_PROW_DRIVER_CANARY is set. +# Only valid if CSI_PROW_DRIVER_CANARY == "canary". configvar CSI_PROW_DRIVER_CANARY_REGISTRY "gcr.io/k8s-staging-sig-storage" "registry for canary images" # The E2E testing can come from an arbitrary repo. The expectation is that @@ -697,7 +697,11 @@ install_csi_driver () { fi if [ "${CSI_PROW_DRIVER_CANARY}" != "stable" ]; then + if [ "${CSI_PROW_DRIVER_CANARY}" == "canary" ]; then images="$images IMAGE_TAG=${CSI_PROW_DRIVER_CANARY} IMAGE_REGISTRY=${CSI_PROW_DRIVER_CANARY_REGISTRY}" + else + images="$images IMAGE_TAG=${CSI_PROW_DRIVER_CANARY}" + fi fi # Ignore: Double quote to prevent globbing and word splitting. # It's intentional here for $images.