Skip to content

Commit 7d5bd20

Browse files
TotktonadaTotktonada
committed
travis-ci: deploy RPM / Deb packages
Now php-tarantool packages are deployed into two kinds of repositories: packagecloud.io and to S3 based ones. The download.tarantool.org now points to the latter, but we keep packagecloud.io for a while to don't break users, which use it directly (we don't recommend it though). See [1] for more info about deprecation of our packagecloud.io repositories. The deployment scripts allows to provide two configurations: production and staging. The former is for deployment from the master branch and from a git tag. The latter is for deployments from a developer branch. It is useful when something should be verified in a specific environment using a built package or when the deployment process itself was modified and should be tested. The difference between production and staging deployment process is how duplicate package versions are handled. It give an error for production deployment, but does not for staging. The new package is discarded in the case for packagecloud.io, but it replaces the old package in S3 repositories. Read comments in the deployment scripts for more details. [1]: tarantool/tarantool#4947 Fixes #117
1 parent 21f638b commit 7d5bd20

8 files changed

+365
-0
lines changed

‎.gitignore

+3
Original file line numberDiff line numberDiff line change
@@ -46,3 +46,6 @@ debian/php5-tarantool.substvars
4646
debian/php5-tarantool/
4747
build
4848
.*.sw[a-z]
49+
50+
# Unencrypted private GPG keys for deployment.
51+
.travis/*.asc

‎.travis.yml

+43
Original file line numberDiff line numberDiff line change
@@ -136,3 +136,46 @@ script:
136136
else
137137
exit 1
138138
fi
139+
140+
deploy:
141+
skip_cleanup: true
142+
provider: script
143+
# Don't skip tagged commits.
144+
#
145+
# The script below will skip deployment when appropriate.
146+
# See the link below for better description why it is necessary.
147+
#
148+
# https://github.com/tarantool/tarantool/issues/3745
149+
on:
150+
all_branches: true
151+
script:
152+
- |
153+
# Skip deployment when it is not expected.
154+
if [ -z "${OS}" ] || [ -z "${DIST}" ]; then
155+
echo "Skip deployment: it is pure testing job w/o any RPM / Deb artefacts"
156+
exit 0
157+
fi
158+
if [ "${TRAVIS_REPO_SLUG}" != "tarantool/tarantool-php" ]; then
159+
echo "Skip deployment: it is a fork, not the base repository"
160+
exit 0
161+
fi
162+
if [ "${TRAVIS_EVENT_TYPE}" = "push" ]; then
163+
echo "Skip deployment: event is not 'push': ${TRAVIS_EVENT_TYPE}"
164+
exit 0
165+
fi
166+
167+
# Choose destination to push packages.
168+
if [ "${TRAVIS_BRANCH}" == "master" ] || [ -n "${TRAVIS_TAG}" ]; then
169+
echo "Set production deployment parameters"
170+
configuration=production
171+
else
172+
echo "Set staging deployment parameters"
173+
configuration=staging
174+
fi
175+
176+
# Deploy to packagecloud repositories.
177+
./.travis/deploy_packagecloud.sh ${configuration}
178+
179+
# Deploy to S3 based repositories.
180+
./.travis/deploy_s3_dependencies.sh
181+
./.travis/deploy_s3.sh ${configuration}

‎.travis/deploy_packagecloud.sh

+120
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
#!/bin/sh
2+
3+
# Deploy to packagecloud repositories
4+
# -----------------------------------
5+
#
6+
# `deploy_packagecloud.sh` is equivalent to
7+
# `deploy_packagecloud.sh staging`.
8+
#
9+
# `deploy_packagecloud.sh staging` requires the following
10+
# environment variables:
11+
#
12+
# - OS
13+
# - DIST
14+
# - DEPLOY_STAGING_PACKAGECLOUD_USER
15+
# - DEPLOY_STAGING_PACKAGECLOUD_TOKEN
16+
#
17+
# `deploy_packagecloud.sh production` requires the following
18+
# environment variables:
19+
#
20+
# - OS
21+
# - DIST
22+
# - DEPLOY_PRODUCTION_PACKAGECLOUD_USER
23+
# - DEPLOY_PRODUCTION_PACKAGECLOUD_TOKEN
24+
#
25+
# If one of those variables is not set or empty, then deployment
26+
# will be skipped.
27+
28+
# Make shell strictier.
29+
#
30+
# - Exit with a failure on a first failed command.
31+
# - Exit with a failure on an attempt to use an unset variable.
32+
# - Print each executed commmand.
33+
#
34+
# Note: The script expects that Travis-CI will filter sensitive
35+
# information (such as a token): 'Display value in build log'
36+
# toogle should be OFF for to keep a value secure.
37+
set -eux
38+
39+
configuration=${1:-staging}
40+
41+
# Choose credentials.
42+
if [ ${configuration} = staging ]; then
43+
DEPLOY_PACKAGECLOUD_USER="${DEPLOY_STAGING_PACKAGECLOUD_USER:-}"
44+
DEPLOY_PACKAGECLOUD_TOKEN="${DEPLOY_STAGING_PACKAGECLOUD_TOKEN:-}"
45+
elif [ ${configuration} = production ]; then
46+
DEPLOY_PACKAGECLOUD_USER="${DEPLOY_PRODUCTION_PACKAGECLOUD_USER:-}"
47+
DEPLOY_PACKAGECLOUD_TOKEN="${DEPLOY_PRODUCTION_PACKAGECLOUD_TOKEN:-}"
48+
else
49+
echo "Unknown configuration: ${configuration}"
50+
exit 1
51+
fi
52+
53+
# Skip deployment if some variables are not set or empty.
54+
if [ -z "${OS:-}" ] || [ -z "${DIST:-}" ] || \
55+
[ -z "${DEPLOY_PACKAGECLOUD_USER}" ] || \
56+
[ -z "${DEPLOY_PACKAGECLOUD_TOKEN}" ]; then
57+
echo "Skip deployment: some of necessary environment"
58+
echo "variables are not set or empty"
59+
exit 0
60+
fi
61+
62+
# Verify that packpack is cloned into the current directory.
63+
packagecloud_tool=./packpack/tools/packagecloud
64+
if [ ! -f "${packagecloud_tool}" ]; then
65+
echo "Could not find ${packagecloud_tool}"
66+
exit 1
67+
fi
68+
69+
# Staging repository: keep older packages in case of a
70+
# version clash.
71+
#
72+
# It would be better to replace old ones, but there is no
73+
# such option in the packagecloud tool we use. It may be
74+
# important if we'll have some manual or automatic testing
75+
# upward a staging repository. But at least CI will not fail
76+
# because a package is already exists.
77+
push_args=""
78+
if [ "${configuration}" = staging ]; then
79+
push_args="${push_args} --ignore-duplicates"
80+
fi
81+
82+
# Setup environment variables for the packagecloud tool.
83+
export PACKAGECLOUD_TOKEN="${DEPLOY_PACKAGECLOUD_TOKEN}"
84+
85+
# We have tarantool repositories on packagecloud.io up to
86+
# 2_4. The next ones present only in the S3 based storage.
87+
for repo in 1_6 1_7 1_9 1_10 2x 2_2 2_3 2_4; do
88+
# FIXME: Enable *.ddeb when packagecloud.io will support it.
89+
for file in build/*.{rpm,deb,dsc}; do
90+
extension=${file##*.}
91+
92+
# Skip non-matched globs: say, build/*.rpm on Debian.
93+
basename="$(basename "${file}" ".${extension}")"
94+
[ "${basename}" = "*" ] && continue
95+
96+
# Push all source files listed in .dsc file together with
97+
# the file.
98+
#
99+
# FIXME: It seems logical to move this logic to the
100+
# packagecloud tool we use.
101+
files="${file}"
102+
if [ "${extension}" = "dsc" ]; then
103+
parse_dsc_file='{
104+
if ($0 == "Files:") {
105+
FILES_SECTION = 1;
106+
} else if (FILES_SECTION != 0) {
107+
print "build/"$3;
108+
}
109+
}'
110+
files="${files} $(awk "${parse_dsc_file}" "${file}")"
111+
fi
112+
113+
# FIXME: The tool fetches distributions.json each time.
114+
# It can cache the data somewhere and reuse during some
115+
# time period until expiration.
116+
user=${DEPLOY_PACKAGECLOUD_USER}
117+
${packagecloud_tool} push ${push_args} ${user}/${repo} ${extension} \
118+
${OS} ${DIST} ${files}
119+
done
120+
done

‎.travis/deploy_production_s3_gpg_private_key.asc.enc

5.39 KB
Binary file not shown.

‎.travis/deploy_s3.sh

+160
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
#!/bin/sh
2+
3+
# Deploy to S3 based repositories
4+
# -------------------------------
5+
#
6+
# `deploy_s3.sh` is equivalent to `deploy_s3.sh staging`.
7+
#
8+
# `deploy_s3.sh staging` requires the following environment
9+
# variables:
10+
#
11+
# - OS
12+
# - DIST
13+
# - DEPLOY_STAGING_S3_ENDPOINT_URL="https://..."
14+
# - DEPLOY_STAGING_S3_LIVE_DIR="s3://my_bucket/foo/bar/live"
15+
# - DEPLOY_STAGING_S3_RELEASE_DIR="s3://my_bucket/foo/bar/release"
16+
# - DEPLOY_STAGING_S3_ACCESS_KEY_ID
17+
# - DEPLOY_STAGING_S3_SECRET_ACCESS_KEY
18+
# - DEPLOY_STAGING_S3_GPG_KEY_FILE_KEY (32 bytes in hex)
19+
# - DEPLOY_STAGING_S3_GPG_KEY_FILE_IV (16 bytes in hex)
20+
#
21+
# `deploy_s3.sh production` requires the following environment
22+
# variables:
23+
#
24+
# - OS
25+
# - DIST
26+
# - DEPLOY_PRODUCTION_S3_ENDPOINT_URL="https://..."
27+
# - DEPLOY_PRODUCTION_S3_LIVE_DIR="s3://my_bucket/foo/bar/live"
28+
# - DEPLOY_PRODUCTION_S3_RELEASE_DIR="s3://my_bucket/foo/bar/release"
29+
# - DEPLOY_PRODUCTION_S3_ACCESS_KEY_ID
30+
# - DEPLOY_PRODUCTION_S3_SECRET_ACCESS_KEY
31+
# - DEPLOY_PRODUCTION_S3_GPG_KEY_FILE_KEY (32 bytes in hex)
32+
# - DEPLOY_PRODUCTION_S3_GPG_KEY_FILE_IV (16 bytes in hex)
33+
#
34+
# If one of those variables is not set or empty, then deployment
35+
# will be skipped.
36+
37+
# Make shell strictier.
38+
#
39+
# - Exit with a failure on a first failed command.
40+
# - Exit with a failure on an attempt to use an unset variable.
41+
# - Print each executed commmand.
42+
#
43+
# Note: The script expects that Travis-CI will filter sensitive
44+
# information (such as a token): 'Display value in build log'
45+
# toogle should be OFF for to keep a value secure.
46+
set -eux
47+
48+
configuration=${1:-staging}
49+
50+
# Choose URLs, directories, keys and so.
51+
if [ ${configuration} = staging ]; then
52+
DEPLOY_S3_ENDPOINT_URL="${DEPLOY_STAGING_S3_ENDPOINT_URL:-}"
53+
DEPLOY_S3_LIVE_DIR="${DEPLOY_STAGING_S3_LIVE_DIR:-}"
54+
DEPLOY_S3_RELEASE_DIR="${DEPLOY_STAGING_S3_RELEASE_DIR:-}"
55+
DEPLOY_S3_ACCESS_KEY_ID="${DEPLOY_STAGING_S3_ACCESS_KEY_ID:-}"
56+
DEPLOY_S3_SECRET_ACCESS_KEY="${DEPLOY_STAGING_S3_SECRET_ACCESS_KEY:-}"
57+
DEPLOY_S3_GPG_KEY_FILE_KEY="${DEPLOY_STAGING_S3_GPG_KEY_FILE_KEY:-}"
58+
DEPLOY_S3_GPG_KEY_FILE_IV="${DEPLOY_STAGING_S3_GPG_KEY_FILE_IV:-}"
59+
elif [ ${configuration} = production ]; then
60+
DEPLOY_S3_ENDPOINT_URL="${DEPLOY_PRODUCTION_S3_ENDPOINT_URL:-}"
61+
DEPLOY_S3_LIVE_DIR="${DEPLOY_PRODUCTION_S3_LIVE_DIR:-}"
62+
DEPLOY_S3_RELEASE_DIR="${DEPLOY_PRODUCTION_S3_RELEASE_DIR:-}"
63+
DEPLOY_S3_ACCESS_KEY_ID="${DEPLOY_PRODUCTION_S3_ACCESS_KEY_ID:-}"
64+
DEPLOY_S3_SECRET_ACCESS_KEY="${DEPLOY_PRODUCTION_S3_SECRET_ACCESS_KEY:-}"
65+
DEPLOY_S3_GPG_KEY_FILE_KEY="${DEPLOY_PRODUCTION_S3_GPG_KEY_FILE_KEY:-}"
66+
DEPLOY_S3_GPG_KEY_FILE_IV="${DEPLOY_PRODUCTION_S3_GPG_KEY_FILE_IV:-}"
67+
else
68+
echo "Unknown configuration: ${configuration}"
69+
exit 1
70+
fi
71+
72+
# Skip deployment if some variables are not set or empty.
73+
if [ -z "${OS:-}" ] || [ -z "${DIST:-}" ] || \
74+
[ -z "${DEPLOY_S3_ENDPOINT_URL}" ] || \
75+
[ -z "${DEPLOY_S3_LIVE_DIR}" ] || \
76+
[ -z "${DEPLOY_S3_RELEASE_DIR}" ] || \
77+
[ -z "${DEPLOY_S3_ACCESS_KEY_ID}" ] || \
78+
[ -z "${DEPLOY_S3_SECRET_ACCESS_KEY}" ] || \
79+
[ -z "${DEPLOY_S3_GPG_KEY_FILE_KEY}" ] || \
80+
[ -z "${DEPLOY_S3_GPG_KEY_FILE_IV}" ]; then
81+
echo "Skip deployment: some of necessary environment"
82+
echo "variables are not set or empty"
83+
exit 0
84+
fi
85+
86+
# Download the tool to deploy to an S3 based repository.
87+
ref=f84cb1aae3144f5677feacf6be31bd4f15e91c2d
88+
base_url="https://raw.githubusercontent.com/tarantool/tarantool/${ref}"
89+
curl -Ssfo update_repo.sh "${base_url}/tools/update_repo.sh"
90+
chmod a+x update_repo.sh
91+
92+
# FIXME: Upstream the patch.
93+
patch -p1 -i .travis/update-repo-sh-use-right-gpg-key.patch
94+
95+
# Decrypt a GPG key.
96+
gpg_key_file=".travis/deploy_${configuration}_s3_gpg_private_key.asc"
97+
openssl aes-256-cbc -K "${DEPLOY_S3_GPG_KEY_FILE_KEY}" \
98+
-iv "${DEPLOY_S3_GPG_KEY_FILE_IV}" -in "${gpg_key_file}.enc" \
99+
-out "${gpg_key_file}" -d
100+
101+
# Import GPG key for signing repository files.
102+
gpg --import --batch "${gpg_key_file}"
103+
104+
# Extract GPG key id for signing repository files.
105+
export GPG_SIGN_KEY="$(gpg --with-colons --import-options show-only \
106+
--import "${gpg_key_file}" | grep ^sec: | cut -d: -f5)"
107+
108+
# Setup environment variables for the update_repo.sh tool.
109+
export AWS_S3_ENDPOINT_URL="${DEPLOY_S3_ENDPOINT_URL}"
110+
export AWS_ACCESS_KEY_ID="${DEPLOY_S3_ACCESS_KEY_ID}"
111+
export AWS_SECRET_ACCESS_KEY="${DEPLOY_S3_SECRET_ACCESS_KEY}"
112+
113+
# ${product} value may affect location of *.deb, *.rpm and related
114+
# files relative to a base repository URL. We can provide it or
115+
# miss: the script will generate correct repository metainfo
116+
# anyway.
117+
#
118+
# However providing meaningful value for this option enables
119+
# grouping of related set of packages into a subdirectory named as
120+
# ${product} (only for Deb repositories at moment of writing
121+
# this).
122+
#
123+
# It is enabled here for consistency with locations of other Deb
124+
# packages in our repositories, but in fact it is the internal
125+
# detail, which does not lead to any change in the user
126+
# experience.
127+
product=php-tarantool
128+
129+
# Setup arguments that are common for all repositories
130+
# (1.10, 2.1, ...).
131+
update_repo_args="--os=${OS} --distribution=${DIST} --product=${product}"
132+
133+
# Staging repository: rewrite a package if there is a previous one
134+
# of the same version.
135+
#
136+
# Note: It differs from a logic in deploy_packagecloud.sh.
137+
if [ "${configuration}" = staging ]; then
138+
update_repo_args="${update_repo_args} --force"
139+
fi
140+
141+
# Deploy to S3 based repositories.
142+
for repo in 1.10 2.1 2.2 2.3 2.4 2.5; do
143+
# Note: The update_repo.sh tool automatically find
144+
# *.{rpm,deb,dsc} within a passed directory, so we just
145+
# pass the directory name: 'build'.
146+
147+
# FIXME: Machine-local locking that is used in the
148+
# update_repo.sh tool is insufficient when we deploy from a
149+
# just created virtual machine.
150+
151+
# Deploy to live repository (per-push).
152+
bucket="${DEPLOY_S3_LIVE_DIR}/${repo}"
153+
./update_repo.sh ${update_repo_args} --bucket="${bucket}" build
154+
155+
# Deploy to release repository (tagged commits).
156+
if [ -n "${TRAVIS_TAG:-}" ]; then
157+
bucket="${DEPLOY_S3_RELEASE_DIR}/${repo}"
158+
./update_repo.sh ${update_repo_args} --bucket="${bucket}" build
159+
fi
160+
done

‎.travis/deploy_s3_dependencies.sh

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
#!/bin/sh
2+
3+
# Prevent procmail package from asking configuration parameters
4+
# interactively.
5+
# See https://github.com/packpack/packpack/issues/7
6+
export DEBIAN_FRONTEND=noninteractive
7+
8+
apt-get install -y procmail # for lockfile tool
9+
apt-get install -y awscli
10+
apt-get install -y reprepro

‎.travis/deploy_staging_s3_gpg_private_key.asc.enc

5.39 KB
Binary file not shown.

‎.travis/update-repo-sh-use-right-gpg-key.patch

+29
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
--- a/update_repo.sh 2020-06-09 17:35:03.332961335 +0300
2+
+++ b/update_repo.sh 2020-06-23 02:26:55.532653673 +0300
3+
@@ -415,7 +415,7 @@
4+
done
5+
# resign the selfsigned InRelease file
6+
$rm_file InRelease
7+
- gpg --clearsign -o InRelease Release
8+
+ gpg -u $GPG_SIGN_KEY --clearsign -o InRelease Release
9+
# resign the Release file
10+
$rm_file Release.gpg
11+
gpg -u $GPG_SIGN_KEY -abs -o Release.gpg Release
12+
@@ -784,7 +784,7 @@
13+
EOF
14+
done
15+
tail -n 1 repodata.adding/repomd.xml >>repodata/repomd.xml
16+
- gpg --detach-sign --armor repodata/repomd.xml
17+
+ gpg -u $GPG_SIGN_KEY --detach-sign --armor repodata/repomd.xml
18+
19+
# copy the packages to S3
20+
for file in $pack_rpms ; do
21+
@@ -901,7 +901,7 @@
22+
tail -n 1 repomd_saved.xml >>repomd.xml
23+
rm -f repomd_saved.xml repomd.xml.asc
24+
popd
25+
- gpg --detach-sign --armor repodata/repomd.xml
26+
+ gpg -u $GPG_SIGN_KEY --detach-sign --armor repodata/repomd.xml
27+
28+
# update the metadata at the S3
29+
$aws_sync_public repodata "$bucket_path/$repopath/repodata"

0 commit comments

Comments
 (0)