Skip to content

DEPS: Unifying testing and building dependencies across builds #29678

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Nov 21, 2019
Merged

DEPS: Unifying testing and building dependencies across builds #29678

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
6a954f2
DEPS: Unifying testing and building dependencies across builds
datapythonista Nov 17, 2019
a6820c7
using pip instead of conda for cython in 32 bits
datapythonista Nov 17, 2019
1bc38f3
Adding pytest-cov, deleted by mistake
datapythonista Nov 17, 2019
492736b
Addressing PR comments
datapythonista Nov 17, 2019
a455fe3
Testing with minimum cython and pytest
datapythonista Nov 21, 2019
80db7bc
Merge remote-tracking branch 'upstream/master' into pytest_versions
datapythonista Nov 21, 2019
a7aa71d
Upgrading pytest, and editing 3.8 build
datapythonista Nov 21, 2019
1692d3c
pytest from pip for 32 bits (new required version not in conda)
datapythonista Nov 21, 2019
01dfdf3
Sync pip deps
datapythonista Nov 21, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 12 additions & 10 deletions ci/deps/azure-36-32bit.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,21 +3,23 @@ channels:
- defaults
- conda-forge
dependencies:
- python=3.6.*
# tools
# - cython=0.29.13 ### Cython 0.29.13 for 32 bits is not available with conda, installing below with pip instead
- pytest
- pytest-xdist>=1.21
- pytest-mock
- hypothesis>=3.58.0
- pytest-azurepipelines
# pandas dependencies
- attrs=19.1.0
- gcc_linux-32
- gcc_linux-32
- gxx_linux-32
- numpy=1.14.*
- python-dateutil
- python=3.6.*
- pytz=2017.2
# universal
- pytest
- pytest-xdist
- pytest-mock
- pytest-azurepipelines
- hypothesis>=3.58.0

# see comment above
- pip
- pip:
# Anaconda doesn't build a new enough Cython
- cython>=0.29.13
- cython==0.29.13
17 changes: 9 additions & 8 deletions ci/deps/azure-36-locale.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,28 +3,29 @@ channels:
- defaults
- conda-forge
dependencies:
- python=3.6.*
# tools
- cython=0.29.13
- pytest
- pytest-xdist>=1.21
- pytest-mock
- hypothesis>=3.58.0
- pytest-azurepipelines
# pandas dependencies
- beautifulsoup4==4.6.0
- bottleneck=1.2.*
- cython=0.29.13
- lxml
- matplotlib=2.2.2
- numpy=1.14.*
- openpyxl=2.4.8
- python-dateutil
- python-blosc
- python=3.6.*
- pytz=2017.2
- scipy
- sqlalchemy=1.1.4
- xlrd=1.1.0
- xlsxwriter=0.9.8
- xlwt=1.2.0
# universal
- pytest>=5.0.0
- pytest-xdist>=1.29.0
- pytest-mock
- pytest-azurepipelines
- hypothesis>=3.58.0
- pip
- pip:
- html5lib==1.0b2
19 changes: 9 additions & 10 deletions ci/deps/azure-36-locale_slow.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,16 @@ channels:
- defaults
- conda-forge
dependencies:
- python=3.6.*
# tools
- cython=0.29.13
- pytest
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need a min pin on pytest (for all .yaml)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've got all these versions in the builds:

  • pytest
  • pytest>=4.0.2
  • pytest>=4.0.2,<5.0
  • pytest=4.5.0
  • pytest>=5.0.0
  • pytest>=5.0.1

Any preference on which you want to require? pytest>-5.0.1 is ok?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah ok with the >=4.0.2 I think < 4 we don't support at all

- pytest-xdist>=1.21
- pytest-mock
- hypothesis>=3.58.0
- pytest-azurepipelines
# pandas dependencies
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you put a blank line before the comments

- beautifulsoup4
- cython>=0.29.13
- gcsfs
- html5lib
- ipython
Expand All @@ -17,20 +25,11 @@ dependencies:
- openpyxl
- pytables
- python-dateutil
- python=3.6.*
- pytz
- s3fs
- scipy
- xarray
- xlrd
- xlsxwriter
- xlwt
# universal
- pytest>=4.0.2
- pytest-xdist
- pytest-mock
- pytest-azurepipelines
- moto
- pip
- pip:
- hypothesis>=3.58.0
17 changes: 9 additions & 8 deletions ci/deps/azure-36-minimum_versions.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,25 +3,26 @@ channels:
- defaults
- conda-forge
dependencies:
- python=3.6.1
# tools
- cython=0.29.13
- pytest
- pytest-xdist>=1.21
- pytest-mock
- hypothesis>=3.58.0
- pytest-azurepipelines
# pandas dependencies
- beautifulsoup4=4.6.0
- bottleneck=1.2.1
- cython>=0.29.13
- jinja2=2.8
- numexpr=2.6.2
- numpy=1.13.3
- openpyxl=2.4.8
- pytables=3.4.2
- python-dateutil=2.6.1
- python=3.6.1
- pytz=2017.2
- scipy=0.19.0
- xlrd=1.1.0
- xlsxwriter=0.9.8
- xlwt=1.2.0
# universal
- html5lib=1.0.1
- hypothesis>=3.58.0
- pytest=4.5.0
- pytest-xdist
- pytest-mock
- pytest-azurepipelines
19 changes: 9 additions & 10 deletions ci/deps/azure-37-locale.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,16 @@ channels:
- defaults
- conda-forge
dependencies:
- python=3.7.*
# tools
- cython=0.29.13
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these should be >=

- pytest
- pytest-xdist>=1.21
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually looking at some of our builds, I think we need pytest-xdist>1.29, which likely implies pytest>5.0.1, if you can confirm this, we can move the pytest pin.

I think that we are likely installing the latest pytest which is fine; these are just a min pin. So on the 36-min versions, can you pin exactly these versions to test.

Copy link
Member Author

@datapythonista datapythonista Nov 17, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in all our builds we have pytest-xdist>=1.21, happy to force 1.29 and pytest>5.0.1 but I don't think it's needed.

I agree with your previous comments. I think we do care about the versions. My point was that I don't think we care about supporting old versions, and we should probably just use the latest. Probably not for black. It's already quite annoying to use it, and if at every new version it reformats the code, we should probably pin it and update it once per year.

- pytest-mock
- hypothesis>=3.58.0
- pytest-azurepipelines
# pandas dependencies
- beautifulsoup4
- cython>=0.29.13
- html5lib
- ipython
- jinja2
Expand All @@ -17,19 +25,10 @@ dependencies:
- openpyxl
- pytables
- python-dateutil
- python=3.7.*
- pytz
- s3fs
- scipy
- xarray
- xlrd
- xlsxwriter
- xlwt
# universal
- pytest>=5.0.1
- pytest-xdist>=1.29.0
- pytest-mock
- pytest-azurepipelines
- pip
- pip:
- hypothesis>=3.58.0
15 changes: 7 additions & 8 deletions ci/deps/azure-37-numpydev.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,19 @@ channels:
- defaults
dependencies:
- python=3.7.*
- pytz
- Cython>=0.29.13
# universal
# pytest < 5 until defaults has pytest-xdist>=1.29.0
- pytest>=4.0.2,<5.0
- pytest-xdist
# tools
- cython=0.29.13
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same (and all others except for the 3.6-min build can have >= for cython)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the minimum versions build we have a >=0.29.13 now. There is a =0.29.13 in another build, not sure if that was intentional.

I guess if we pin cython, when this causes a failure, the immediate action will be to pin it to something bigger where it works. So, is it really worth forcing in a build the 0.29.3 version?

- pytest
- pytest-xdist>=1.21
- pytest-mock
- hypothesis>=3.58.0
- pytest-azurepipelines
# pandas dependencies
- pytz
- pip
- pip:
- "git+git://github.com/dateutil/dateutil.git"
- "-f https://7933911d6844c6c53a7d-47bd50c35cd79bd838daf386af554a83.ssl.cf2.rackcdn.com"
- "--pre"
- "numpy"
- "scipy"
# https://github.com/pandas-dev/pandas/issues/27421
- pytest-azurepipelines<1.0.0
19 changes: 9 additions & 10 deletions ci/deps/azure-macos-36.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,15 @@ name: pandas-dev
channels:
- defaults
dependencies:
- python=3.6.*
# tools
- cython=0.29.13
- pytest
- pytest-xdist>=1.21
- pytest-mock
- hypothesis>=3.58.0
- pytest-azurepipelines
# pandas dependencies
- beautifulsoup4
- bottleneck
- html5lib
Expand All @@ -14,7 +23,6 @@ dependencies:
- openpyxl
- pyarrow
- pytables
- python=3.6.*
- python-dateutil==2.6.1
- pytz
- xarray
Expand All @@ -23,13 +31,4 @@ dependencies:
- xlwt
- pip
- pip:
# Anaconda / conda-forge don't build for 3.5
- cython>=0.29.13
- pyreadstat
# universal
- pytest>=5.0.1
- pytest-xdist>=1.29.0
- pytest-mock
- hypothesis>=3.58.0
# https://github.com/pandas-dev/pandas/issues/27421
- pytest-azurepipelines<1.0.0
17 changes: 9 additions & 8 deletions ci/deps/azure-windows-36.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,15 @@ channels:
- conda-forge
- defaults
dependencies:
- python=3.6.*
# tools
- cython=0.29.13
- pytest
- pytest-xdist>=1.21
- pytest-mock
- hypothesis>=3.58.0
- pytest-azurepipelines
# pandas dependencies
- blosc
- bottleneck
- fastparquet>=0.2.1
Expand All @@ -13,16 +22,8 @@ dependencies:
- pyarrow
- pytables
- python-dateutil
- python=3.6.*
- pytz
- scipy
- xlrd
- xlsxwriter
- xlwt
# universal
- cython>=0.29.13
- pytest>=5.0.1
- pytest-xdist>=1.29.0
- pytest-mock
- pytest-azurepipelines
- hypothesis>=3.58.0
17 changes: 9 additions & 8 deletions ci/deps/azure-windows-37.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,15 @@ channels:
- defaults
- conda-forge
dependencies:
- python=3.7.*
# tools
- cython=0.29.13
- pytest
- pytest-xdist>=1.21
- pytest-mock
- hypothesis>=3.58.0
- pytest-azurepipelines
# pandas dependencies
- beautifulsoup4
- bottleneck
- gcsfs
Expand All @@ -15,7 +24,6 @@ dependencies:
- numpy=1.14.*
- openpyxl
- pytables
- python=3.7.*
- python-dateutil
- pytz
- s3fs
Expand All @@ -24,11 +32,4 @@ dependencies:
- xlrd
- xlsxwriter
- xlwt
# universal
- cython>=0.29.13
- pytest>=5.0.0
- pytest-xdist>=1.29.0
- pytest-mock
- pytest-azurepipelines
- hypothesis>=3.58.0
- pyreadstat
16 changes: 9 additions & 7 deletions ci/deps/travis-36-cov.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,15 @@ channels:
- defaults
- conda-forge
dependencies:
- python=3.6.*
# tools
- cython=0.29.13
- pytest
- pytest-xdist>=1.21
- pytest-mock
- hypothesis>=3.58.0
- pytest-cov # this is only needed in the coverage build
# pandas dependencies
- beautifulsoup4
- botocore>=1.11
- cython>=0.29.13
Expand All @@ -27,7 +36,6 @@ dependencies:
- pymysql
- pytables
- python-snappy
- python=3.6.*
- pytz
- s3fs
- scikit-learn
Expand All @@ -38,12 +46,6 @@ dependencies:
- xlrd
- xlsxwriter
- xlwt
# universal
- pytest>=5.0.1
- pytest-xdist>=1.29.0
- pytest-cov
- pytest-mock
- hypothesis>=3.58.0
- pip
- pip:
- brotlipy
Expand Down
17 changes: 8 additions & 9 deletions ci/deps/travis-36-locale.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,17 @@ channels:
- defaults
- conda-forge
dependencies:
- python=3.6.*
# tools
- cython=0.29.13
- pytest
- pytest-xdist>=1.21
- pytest-mock
- hypothesis>=3.58.0
# pandas dependencies
- beautifulsoup4
- blosc=1.14.3
- python-blosc
- cython>=0.29.13
- fastparquet=0.2.1
- gcsfs=0.2.2
- html5lib
Expand All @@ -24,7 +31,6 @@ dependencies:
- pymysql=0.7.11
- pytables
- python-dateutil
- python=3.6.*
- pytz
- s3fs=0.3.0
- scipy
Expand All @@ -33,10 +39,3 @@ dependencies:
- xlrd
- xlsxwriter
- xlwt
# universal
- pytest>=5.0.1
- pytest-xdist>=1.29.0
- pytest-mock
- pip
- pip:
- hypothesis>=3.58.0
Loading