Skip to content

Commit 464f20e

Browse files
humitoshumitos
authored
Build: do not upload build.tool to production S3 (#9098)
* Build: do not upload `build.tool` to production S3 Avoid uploading the .tar.gz to production S3 since we will be managing this via CircleCI and an AWS orbs. See readthedocs/readthedocs-docker-images#179 We need to keep the upload part for development environment, tho. * Build: join TOOL and VERSION with a `-` to simplify CircleCI routine * Do not delete .tar.gz from local host when on production * Remove old documentation We are not going to run this command in production by hand anymore. However, in case it's required, I'm linking to the issue that explains how to do it. * Skip uploading when running from inside CIRCLECI * Split docker iamge name from full to just keep the name without date * Revert "Split docker iamge name from full to just keep the name without date" This reverts commit e08a7de. * Receive arguments as previously * Comment about pinning transitive dependencies * Build Python 3.6.15 using `clang`
1 parent 853f3fc commit 464f20e

File tree

1 file changed

+37
-60
lines changed

1 file changed

+37
-60
lines changed

‎scripts/compile_version_upload_s3.sh

+37-60
Original file line numberDiff line numberDiff line change
@@ -23,52 +23,16 @@
2323
# PRODUCTION ENVIRONMENT
2424
#
2525
# To create a pre-compiled cached version and make it available on production,
26-
# **the script must be ran from a builder (build-default or build-large)** and
27-
# it's required to set several environment variables for an IAM user with
28-
# permissions on ``readthedocs(inc)-build-tools-prod`` S3's bucket. Also, note
29-
# that in production we need to install `aws` Python package to run the script.
30-
# We can do this in a different virtualenv to avoid collision with the builder's
31-
# code.
32-
#
33-
# The whole process would be something like:
34-
#
35-
# ssh util01
36-
# ssh `scaling status -s build-default -q | head -n 1`
37-
#
38-
# sudo su - docs
39-
# TOOL=python
40-
# VERSION=3.10.0
41-
#
42-
# cd /home/docs/checkouts/readthedocs.org/scripts
43-
# virtualenv venv
44-
# source venv/bin/activate
45-
# pip install awscli==1.20.34
46-
#
47-
# export AWS_REGION=...
48-
# export AWS_ACCESS_KEY_ID=...
49-
# export AWS_SECRET_ACCESS_KEY=...
50-
# export AWS_BUILD_TOOLS_BUCKET=readthedocs(inc)-build-tools-prod
51-
#
52-
# ./compile_version_upload.sh $TOOL $VERSION
53-
#
54-
#
55-
# ONE-LINE COMMAND FROM UTIL01 PRODUCTION
56-
#
57-
# TOOL=python
58-
# VERSION=3.10.0
59-
# AWS_BUILD_TOOLS_BUCKET=readthedocs(inc)-build-tools-prod
60-
#
61-
# ssh `scaling status -s build-default -q | head -n 1` \
62-
# "cd /home/docs && \
63-
# sudo -u docs virtualenv --python python3 /home/docs/buildtools && \
64-
# sudo -u docs /home/docs/buildtools/bin/pip install awscli==1.20.34 && \
65-
# sudo -u docs env AWS_REGION=$AWS_REGION \
66-
# AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
67-
# AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
68-
# AWS_BUILD_TOOLS_BUCKET=$AWS_BUILD_TOOLS_BUCKET \
69-
# PATH=/home/docs/buildtools/bin:${PATH} \
70-
# /home/docs/checkouts/readthedocs.org/scripts/compile_version_upload_s3.sh $TOOL $VERSION"
26+
# we use a CircleCI job
27+
# (https://github.com/readthedocs/readthedocs-docker-images/blob/main/.circleci/config.yml)
28+
# It requires to set several environment variables for an IAM user with
29+
# permissions on ``readthedocs(inc)-build-tools-prod`` S3's bucket. These
30+
# variables are defined via the CircleCI UI under the `readthedocs-docker-image`
31+
# project.
7132
#
33+
# Note that if for some reason, you need to run this command *outside CircleCI*
34+
# you can find more information in this comment:
35+
# https://github.com/readthedocs/readthedocs-ops/issues/1155#issuecomment-1082615972
7236
#
7337
# USAGE
7438
#
@@ -90,6 +54,7 @@ set -x # Echo commands
9054
# Define variables
9155
SLEEP=350 # Container timeout
9256
OS="${OS:-ubuntu-22.04}" # Docker image name
57+
9358
TOOL=$1
9459
VERSION=$2
9560

@@ -103,7 +68,16 @@ echo "Running all the commands in Docker container: $CONTAINER_ID"
10368
# Install the tool version requested
10469
if [[ $TOOL == "python" ]]
10570
then
106-
docker exec --env PYTHON_CONFIGURE_OPTS="--enable-shared" $CONTAINER_ID asdf install $TOOL $VERSION
71+
if [[ $VERSION == "3.6.15" ]]
72+
then
73+
# Special command for Python 3.6.15
74+
# See https://github.com/pyenv/pyenv/issues/1889#issuecomment-833587851
75+
docker exec --user root $CONTAINER_ID apt-get update
76+
docker exec --user root $CONTAINER_ID apt-get install --yes clang
77+
docker exec --env PYTHON_CONFIGURE_OPTS="--enable-shared" --env CC=clang $CONTAINER_ID asdf install $TOOL $VERSION
78+
else
79+
docker exec --env PYTHON_CONFIGURE_OPTS="--enable-shared" $CONTAINER_ID asdf install $TOOL $VERSION
80+
fi
10781
else
10882
docker exec $CONTAINER_ID asdf install $TOOL $VERSION
10983
fi
@@ -113,6 +87,10 @@ docker exec $CONTAINER_ID asdf global $TOOL $VERSION
11387
docker exec $CONTAINER_ID asdf reshim $TOOL
11488

11589
# Install dependencies for this version
90+
#
91+
# TODO: pin all transitive dependencies with pip-tools or similar. We can find
92+
# the current versions by running `pip freeze` in production and stick with them
93+
# for now to avoid changing versions.
11694
if [[ $TOOL == "python" ]] && [[ ! $VERSION =~ (^miniconda.*|^mambaforge.*) ]]
11795
then
11896
RTD_PIP_VERSION=21.2.4
@@ -138,22 +116,21 @@ docker cp $CONTAINER_ID:/home/docs/$OS-$TOOL-$VERSION.tar.gz .
138116
# Kill the container
139117
docker container kill $CONTAINER_ID
140118

141-
# Upload the .tar.gz to S3
142-
AWS_ENDPOINT_URL="${AWS_ENDPOINT_URL:-http://localhost:9000}"
143-
AWS_BUILD_TOOLS_BUCKET="${AWS_BUILD_TOOLS_BUCKET:-build-tools}"
144-
AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID:-admin}"
145-
AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY:-password}"
146-
147-
if [[ -z $AWS_REGION ]]
119+
if [[ -z $CIRCLECI ]]
148120
then
149-
# Development environment
121+
# Upload the .tar.gz to S3 development environment
150122
echo "Uploading to dev environment"
123+
124+
AWS_ENDPOINT_URL="${AWS_ENDPOINT_URL:-http://localhost:9000}"
125+
AWS_BUILD_TOOLS_BUCKET="${AWS_BUILD_TOOLS_BUCKET:-build-tools}"
126+
AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID:-admin}"
127+
AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY:-password}"
128+
151129
aws --endpoint-url $AWS_ENDPOINT_URL s3 cp $OS-$TOOL-$VERSION.tar.gz s3://$AWS_BUILD_TOOLS_BUCKET
130+
131+
# Delete the .tar.gz file from the host
132+
rm $OS-$TOOL-$VERSION.tar.gz
152133
else
153-
# Production environment does not requires `--endpoint-url`
154-
echo "Uploading to prod environment"
155-
aws s3 cp $OS-$TOOL-$VERSION.tar.gz s3://$AWS_BUILD_TOOLS_BUCKET
134+
echo "Skip uploading .tar.gz file because it's being run from inside CircleCI."
135+
echo "It should be uploaded by orbs/aws automatically."
156136
fi
157-
158-
# Delete the .tar.gz file from the host
159-
rm $OS-$TOOL-$VERSION.tar.gz

0 commit comments

Comments
 (0)