Skip to content

CI/DOC: Building the documentation with azure-pipelines #26648

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Jun 13, 2019
Merged

CI/DOC: Building the documentation with azure-pipelines #26648

Changes from 10 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
f09761c
CI/DOC: Building the documentation with azure-pipelines
Jun 4, 2019
c959c2c
Synchronizing requirements-dev.txt
Jun 4, 2019
bc8ff7e
Changing dependencies of the doc build in azure
Jun 5, 2019
b2340e3
Merge remote-tracking branch 'upstream/master' into docs_azure
Jun 5, 2019
861e5b1
Reverting changes to dependencies
Jun 5, 2019
0a6f523
Fixing typos, and improving instructions about private/public keys
Jun 5, 2019
ec2587f
Merge branch 'docs_azure' of https://github.com/datapythonista/pandas…
Jun 5, 2019
8072f13
Merge remote-tracking branch 'upstream/master' into docs_azure
Jun 5, 2019
aa079d5
Adding missing parenthesis in comment
Jun 5, 2019
b84a9bf
Merge remote-tracking branch 'upstream/master' into docs_azure
Jun 6, 2019
d9ef920
Merge remote-tracking branch 'upstream/master' into docs_azure
Jun 10, 2019
e51c634
Not failing the build if the push of the docs fails (making the push …
Jun 10, 2019
f31ae50
Leaving the doc build green even if the push fails (removing the fail…
Jun 10, 2019
9a0523a
Removing unused CONDA_ENV variable
datapythonista Jun 12, 2019
7c945fd
Merge branch 'docs_azure' of github.com:datapythonista/pandas into do…
datapythonista Jun 12, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,3 +116,65 @@ jobs:
fi
displayName: 'Running benchmarks'
condition: true

- job: 'Docs'
Copy link
Contributor

@jreback jreback Jun 8, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you make it so if this job fails it won't fail the entire build? just so we don't replicate the same problem as last time

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't seem like pipelines has a feature like the travis one for allowed failures.

I can make the git push step finish green even if it fails, I guess that's the closer option. But this time I set up the CI in my pandas fork, and made sure the part only executed after merging to master is able to push to dev.pandas.io, and it's working correctly.

Let me know if you want to be extra sure, but shouldn't happen the same as last time, the whole job is tested now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right that's the part that could fail (as its only done on master); maybe not fail even if that fails

pool:
vmImage: ubuntu-16.04
timeoutInMinutes: 90
steps:
- script: |
echo '##vso[task.setvariable variable=CONDA_ENV]pandas-dev'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this used for? (I don't find it anywhere used in our CI config)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is used in ci/setup_env.sh to create the conda environment. It's standard in all builds.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any "CONDA_ENV" in https://github.com/pandas-dev/pandas/blob/master/ci/setup_env.sh, it seems this name is hardcoded everywhre (also in this yaml file a few lines below)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you're right, I think it was removed when ci/setup.sh was refactored to be used everywhere and is not needed anymore.

echo '##vso[task.setvariable variable=ENV_FILE]ci/deps/travis-36-doc.yaml'
displayName: 'Setting environment variables'

- script: |
export PATH=$HOME/miniconda3/bin:$PATH
sudo apt-get install -y libc6-dev-i386
ci/setup_env.sh
displayName: 'Setup environment and build pandas'

- script: |
export PATH=$HOME/miniconda3/bin:$PATH
source activate pandas-dev
doc/make.py
displayName: 'Build documentation'

- script: |
cd doc/build/html
git init
touch .nojekyll
echo "dev.pandas.io" > CNAME
git add --all .
git config user.email "[email protected]"
git config user.name "pandas-docs-bot"
git commit -m "pandas documentation in master"
displayName: 'Create git repo for docs build'
condition : |
and(not(eq(variables['Build.Reason'], 'PullRequest')),
eq(variables['Build.SourceBranch'], 'refs/heads/master'))

# For `InstallSSHKey@0` to work, next steps are required:
# 1. Generate a pair of private/public keys (i.e. `ssh-keygen -t rsa -b 4096 -C "[email protected]"`)
# 2. Go to "Library > Secure files" in the Azure Pipelines dashboard: https://dev.azure.com/pandas-dev/pandas/_library?itemType=SecureFiles
# 3. Click on "+ Secure file"
# 4. Upload the private key (the name of the file must match with the specified in "sshKeySecureFile" input below, "pandas_docs_key")
# 5. Click on file name after it is created, tick the box "Authorize for use in all pipelines" and save
# 6. The public key specified in "sshPublicKey" is the pair of the uploaded private key, and needs to be set as a deploy key of the repo where the docs will be pushed (with write access): https://github.com/pandas-dev/pandas-dev.github.io/settings/keys
- task: InstallSSHKey@0
inputs:
hostName: 'github.com,192.30.252.128 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ=='
sshPublicKey: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDHmz3l/EdqrgNxEUKkwDUuUcLv91unig03pYFGO/DMIgCmPdMG96zAgfnESd837Rm0wSSqylwSzkRJt5MV/TpFlcVifDLDQmUhqCeO8Z6dLl/oe35UKmyYICVwcvQTAaHNnYRpKC5IUlTh0JEtw9fGlnp1Ta7U1ENBLbKdpywczElhZu+hOQ892zqOj3CwA+U2329/d6cd7YnqIKoFN9DWT3kS5K6JE4IoBfQEVekIOs23bKjNLvPoOmi6CroAhu/K8j+NCWQjge5eJf2x/yTnIIP1PlEcXoHIr8io517posIx3TBup+CN8bNS1PpDW3jyD3ttl1uoBudjOQrobNnJeR6Rn67DRkG6IhSwr3BWj8alwUG5mTdZzwV5Pa9KZFdIiqX7NoDGg+itsR39QCn0thK8lGRNSR8KrWC1PSjecwelKBO7uQ7rnk/rkrZdBWR4oEA8YgNH8tirUw5WfOr5a0AIaJicKxGKNdMxZt+zmC+bS7F4YCOGIm9KHa43RrKhoGRhRf9fHHHKUPwFGqtWG4ykcUgoamDOURJyepesBAO3FiRE9rLU6ILbB3yEqqoekborHmAJD5vf7PWItW3Q/YQKuk3kkqRcKnexPyzyyq5lUgTi8CxxZdaASIOu294wjBhhdyHlXEkVTNJ9JKkj/obF+XiIIp0cBDsOXY9hDQ== [email protected]'
sshKeySecureFile: 'pandas_docs_key'
displayName: 'Install GitHub ssh deployment key'
condition : |
and(not(eq(variables['Build.Reason'], 'PullRequest')),
eq(variables['Build.SourceBranch'], 'refs/heads/master'))

- script: |
cd doc/build/html
git remote add origin [email protected]:pandas-dev/pandas-dev.github.io.git
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At some point you also had a pandas-dev-docs, that's removed again?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I started using that, but then I removed it. Using <organization-name>.github.io we're able to use directly a custom domain dev.pandas.io. Any other name of repo will make the url something like dev.pandas.io/pandas-dev-docs. Personally I think it'll be very nice to have dev.pandas.io, even more when we move to pandas.io. After almost two years working in the docs I still don't remember the url of the travis docs. :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okido, I am not fully sure about the final url, but that's a discussion for later :-)

git push origin master -f
displayName: 'Publish docs to GitHub pages'
condition : |
and(not(eq(variables['Build.Reason'], 'PullRequest')),
eq(variables['Build.SourceBranch'], 'refs/heads/master'))