CI/DOC: Building the documentation with azure-pipelines #26648

datapythonista · 2019-06-04T16:29:15Z

xref Azure CI followup #22766, CI: build documentation for Windows #26574
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

Looks like the problem in #26591 was that the HostName input of the azure task that installs the key, is not the host name, but the know_hosts line for the host name where to push.

I set up azure-pipelines in my pandas branch, to be able to see the master builds and avoid surprises, and this branch is successfully pushing the docs to https://dev.pandas.io

@TomAugspurger since there were differences between the travis docs dependencies and environment.yml I think it makes more sense to change and use environment.yml here, so we can detect and fix any problem while we still have the travis docs, and then simply remove the travis docs dependencies with the build when we are happy with this.

@jorisvandenbossche the ImportError for pyarrow doesn't seem to be because pyarrow is not in environment.yml, it is. There is something weird with that, need to have a look.

codecov · 2019-06-04T17:05:57Z

Codecov Report

Merging #26648 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #26648      +/-   ##
==========================================
- Coverage   91.87%   91.87%   -0.01%     
==========================================
  Files         174      174              
  Lines       50692    50692              
==========================================
- Hits        46575    46571       -4     
- Misses       4117     4121       +4

Flag	Coverage Δ
#multiple	`90.41% <ø> (ø)`	⬆️
#single	`41.78% <ø> (-0.09%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/gbq.py	`78.94% <0%> (-10.53%)`	⬇️
pandas/core/frame.py	`97% <0%> (-0.12%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c07d71d...c959c2c. Read the comment docs.

codecov · 2019-06-04T17:06:08Z

Codecov Report

Merging #26648 into master will increase coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #26648      +/-   ##
==========================================
+ Coverage   91.71%   91.73%   +0.01%     
==========================================
  Files         178      178              
  Lines       50740    50774      +34     
==========================================
+ Hits        46538    46575      +37     
+ Misses       4202     4199       -3

Flag	Coverage Δ
#multiple	`90.32% <ø> (+0.02%)`	⬆️
#single	`41.19% <ø> (-0.13%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/gbq.py	`78.94% <0%> (-10.53%)`	⬇️
pandas/core/indexes/interval.py	`96.11% <0%> (-0.32%)`	⬇️
pandas/core/frame.py	`96.88% <0%> (-0.12%)`	⬇️
pandas/plotting/_matplotlib/tools.py	`78.4% <0%> (ø)`	⬆️
pandas/core/indexes/datetimes.py	`96.37% <0%> (ø)`	⬆️
pandas/plotting/_matplotlib/style.py	`85.96% <0%> (ø)`	⬆️
pandas/core/indexes/timedeltas.py	`90.96% <0%> (ø)`	⬆️
pandas/plotting/_matplotlib/timeseries.py	`66.32% <0%> (ø)`	⬆️
pandas/core/indexes/datetimelike.py	`98.15% <0%> (ø)`	⬆️
pandas/core/series.py	`93.64% <0%> (+0.02%)`	⬆️
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0f3e8e8...7c945fd. Read the comment docs.

jorisvandenbossche · 2019-06-04T20:27:58Z

@jorisvandenbossche the ImportError for pyarrow doesn't seem to be because pyarrow is not in environment.yml, it is. There is something weird with that, need to have a look.

Ah, yes, indeed. Looking at the error message now, it is indeed a problem with the installation, not that it is not installed.
Possible reason is a mixture of defaults and conda-forge (eg arrow-cpp is from conda-forge, but pyarrow from defaults)

datapythonista · 2019-06-05T09:43:10Z

I think the problem is that pyarrow manages to install without a dependency or with something broken, but then it fails on import. We should probably append the original error message to the error we raise.

ImportError: /home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNK5arrow6Status8ToStringB5cxx11Ev

Any idea on what is missing?

jorisvandenbossche · 2019-06-05T09:58:17Z

I think it is related to the mentioned defaults/conda-forge mixing, but why conda is doing that, not sure ..

datapythonista · 2019-06-05T10:08:27Z

I see, not sure why arrow-cpp is installed from conda-forge with environment.yml. I'm in a windows machine right now, and I get the same, but pyarrow is working. Another difference is that with environment.yml is using Python 3.7 and not 3.6. But again, in my local set up, pyarrow still works with it.

I'll make azure build with travis-36-doc.yaml and see if that fixes the problem, otherwise I'm not sure what else we can do.

datapythonista · 2019-06-05T11:32:22Z

Looks like building with ci/deps/travis-36-doc.yaml doesn't cause the pyarrow problems in azure. Using that here, so we can merge this, and start to build the docs from azure.

I added the missing dependencies in environment.yml in #26657 and I opened #26656 for the pyarrow error.

…into docs_azure

datapythonista · 2019-06-06T09:35:17Z

It's very weird, but the CI was green, and eventually it became red because instead of referencing its own build, was referencing a build of a different PR (https://travis-ci.org/pandas-dev/pandas/jobs/541912722).

I rebased, hope this fix it, but mentioning here in case someone else sees the same, we should probably report the problem to github.

jorisvandenbossche · 2019-06-06T18:51:49Z

The clipboard is not yet working, see here how I recently fixed it on Travis (https://github.com/pandas-dev/pandas/pull/26103/files), but no idea if something similar can fix it on appveyor (might actually also be a good question if the tests for this are working).

datapythonista · 2019-06-07T07:57:50Z

Thanks @jorisvandenbossche I saw the warning, but thought it was one of the warnings that also happens in Travis.

I opened #26700 to address it separately. No change should be required here.

jreback · 2019-06-08T22:00:28Z

azure-pipelines.yml

@@ -116,3 +116,65 @@ jobs:
      fi
    displayName: 'Running benchmarks'
    condition: true
+
+- job: 'Docs'


can you make it so if this job fails it won't fail the entire build? just so we don't replicate the same problem as last time

Doesn't seem like pipelines has a feature like the travis one for allowed failures.

I can make the git push step finish green even if it fails, I guess that's the closer option. But this time I set up the CI in my pandas fork, and made sure the part only executed after merging to master is able to push to dev.pandas.io, and it's working correctly.

Let me know if you want to be extra sure, but shouldn't happen the same as last time, the whole job is tested now.

right that's the part that could fail (as its only done on master); maybe not fail even if that fails

…fail to test if it works)

…ure of the build for testing)

datapythonista · 2019-06-10T13:20:16Z

@jreback made the changes so the CI is still green even if the push of the documentation from azure fails. Tested both that the push works, and that if it fails the CI stays green.

Once merged, this will start updating dev.pandas.io with the master version of the docs.

datapythonista · 2019-06-12T07:51:05Z

@jorisvandenbossche can you have a look here? I think would be good to get this merged some time before we activate the fail on warning in the docs.

jorisvandenbossche

tiny question

jorisvandenbossche · 2019-06-12T08:03:27Z

azure-pipelines.yml

+  timeoutInMinutes: 90
+  steps:
+  - script: |
+      echo '##vso[task.setvariable variable=CONDA_ENV]pandas-dev'


what is this used for? (I don't find it anywhere used in our CI config)

This is used in ci/setup_env.sh to create the conda environment. It's standard in all builds.

I don't see any "CONDA_ENV" in https://github.com/pandas-dev/pandas/blob/master/ci/setup_env.sh, it seems this name is hardcoded everywhre (also in this yaml file a few lines below)

I think you're right, I think it was removed when ci/setup.sh was refactored to be used everywhere and is not needed anymore.

jorisvandenbossche · 2019-06-12T08:14:24Z

azure-pipelines.yml

+
+  - script: |
+      cd doc/build/html
+      git remote add origin [email protected]:pandas-dev/pandas-dev.github.io.git


At some point you also had a pandas-dev-docs, that's removed again?

Yes, I started using that, but then I removed it. Using <organization-name>.github.io we're able to use directly a custom domain dev.pandas.io. Any other name of repo will make the url something like dev.pandas.io/pandas-dev-docs. Personally I think it'll be very nice to have dev.pandas.io, even more when we move to pandas.io. After almost two years working in the docs I still don't remember the url of the travis docs. :)

Okido, I am not fully sure about the final url, but that's a discussion for later :-)

…cs_azure

datapythonista · 2019-06-12T14:25:14Z

@jorisvandenbossche addressed your comments, CI is green

jorisvandenbossche · 2019-06-13T08:54:27Z

OK, let's try again! :)

datapythonista · 2019-06-13T10:07:14Z

thanks @jorisvandenbossche, CI is green, and docs where successfully built and pushed to http://dev.pandas.io/ :)

Marc Garcia added 2 commits June 4, 2019 16:19

CI/DOC: Building the documentation with azure-pipelines

f09761c

Synchronizing requirements-dev.txt

c959c2c

datapythonista added Docs CI Continuous Integration Dependencies Required and optional dependencies labels Jun 4, 2019

Marc Garcia added 2 commits June 5, 2019 11:10

Changing dependencies of the doc build in azure

bc8ff7e

Merge remote-tracking branch 'upstream/master' into docs_azure

b2340e3

datapythonista mentioned this pull request Jun 5, 2019

CI/DOC: pyarrow ImportError when using environment.yml to build the docs #26656

Closed

Reverting changes to dependencies

861e5b1

datapythonista mentioned this pull request Jun 5, 2019

DEPS: Adding missing doc dependencies to environment.yml #26657

Merged

Marc Garcia added 5 commits June 5, 2019 15:56

Fixing typos, and improving instructions about private/public keys

0a6f523

Merge branch 'docs_azure' of https://github.com/datapythonista/pandas …

ec2587f

…into docs_azure

Merge remote-tracking branch 'upstream/master' into docs_azure

8072f13

Adding missing parenthesis in comment

aa079d5

Merge remote-tracking branch 'upstream/master' into docs_azure

b84a9bf

datapythonista mentioned this pull request Jun 7, 2019

DOC: Not running clipboard examples in the doc #26700

Closed

4 tasks

jreback reviewed Jun 8, 2019

View reviewed changes

datapythonista mentioned this pull request Jun 8, 2019

WIP: Azure pipelines configuration astropy/astropy#8445

Closed

4 tasks

Marc Garcia added 2 commits June 10, 2019 11:45

Merge remote-tracking branch 'upstream/master' into docs_azure

d9ef920

Not failing the build if the push of the docs fails (making the push …

e51c634

…fail to test if it works)

Leaving the doc build green even if the push fails (removing the fail…

f31ae50

…ure of the build for testing)

datapythonista changed the title ~~CI/DOC: Building the documentation with azure-pipelines (2nd try)~~ CI/DOC: Building the documentation with azure-pipelines Jun 10, 2019

jorisvandenbossche reviewed Jun 12, 2019

View reviewed changes

datapythonista added 2 commits June 12, 2019 10:02

Removing unused CONDA_ENV variable

9a0523a

Merge branch 'docs_azure' of github.com:datapythonista/pandas into do…

7c945fd

…cs_azure

jorisvandenbossche merged commit e8132a2 into pandas-dev:master Jun 13, 2019

jorisvandenbossche added this to the 0.25.0 milestone Jun 13, 2019

This was referenced Jun 14, 2019

DOC: Remove travis doc build #26856

Merged

DOC: Update development documentation urls #26857

Merged

ghost mentioned this pull request Jul 11, 2019

CI checks doesn't catch commit subject whitespace errors #27345

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI/DOC: Building the documentation with azure-pipelines #26648

CI/DOC: Building the documentation with azure-pipelines #26648

datapythonista commented Jun 4, 2019

codecov bot commented Jun 4, 2019

codecov bot commented Jun 4, 2019 •

edited

Loading

jorisvandenbossche commented Jun 4, 2019

datapythonista commented Jun 5, 2019

jorisvandenbossche commented Jun 5, 2019

datapythonista commented Jun 5, 2019

datapythonista commented Jun 5, 2019

datapythonista commented Jun 6, 2019

jorisvandenbossche commented Jun 6, 2019

datapythonista commented Jun 7, 2019

jreback Jun 8, 2019 •

edited

Loading

datapythonista Jun 8, 2019

jreback Jun 8, 2019

datapythonista commented Jun 10, 2019

datapythonista commented Jun 12, 2019

jorisvandenbossche left a comment

jorisvandenbossche Jun 12, 2019

datapythonista Jun 12, 2019

jorisvandenbossche Jun 12, 2019

datapythonista Jun 12, 2019

jorisvandenbossche Jun 12, 2019

datapythonista Jun 12, 2019

jorisvandenbossche Jun 12, 2019

datapythonista commented Jun 12, 2019

jorisvandenbossche commented Jun 13, 2019

datapythonista commented Jun 13, 2019

CI/DOC: Building the documentation with azure-pipelines #26648

CI/DOC: Building the documentation with azure-pipelines #26648

Conversation

datapythonista commented Jun 4, 2019

codecov bot commented Jun 4, 2019

Codecov Report

codecov bot commented Jun 4, 2019 • edited Loading

Codecov Report

jorisvandenbossche commented Jun 4, 2019

datapythonista commented Jun 5, 2019

jorisvandenbossche commented Jun 5, 2019

datapythonista commented Jun 5, 2019

datapythonista commented Jun 5, 2019

datapythonista commented Jun 6, 2019

jorisvandenbossche commented Jun 6, 2019

datapythonista commented Jun 7, 2019

jreback Jun 8, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

datapythonista commented Jun 10, 2019

datapythonista commented Jun 12, 2019

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

datapythonista commented Jun 12, 2019

jorisvandenbossche commented Jun 13, 2019

datapythonista commented Jun 13, 2019

codecov bot commented Jun 4, 2019 •

edited

Loading

jreback Jun 8, 2019 •

edited

Loading