CI: Test pyarrow nightly instead of intermediate versions #52211
Conversation
| - pytest-xdist>=2.2.0 | ||
| - hypothesis>=6.34.2 | ||
| - pytest-asyncio>=0.17.0 | ||
| - boto3 |
There was a problem hiding this comment.
Is boto3 a required dep for pyarrow?
There was a problem hiding this comment.
Don't think so. I think i accidentally copied this from another deps file
| @@ -92,22 +92,18 @@ def _get_path_or_handle( | |||
| if fs is not None: | |||
| pa_fs = import_optional_dependency("pyarrow.fs", errors="ignore") | |||
| fsspec = import_optional_dependency("fsspec", errors="ignore") | |||
| if pa_fs is None and fsspec is None: | |||
There was a problem hiding this comment.
The build dependency setup uncovered a bug in the recent pyarrow Filesystem implementation not raising a ValueError consistently if pyarrow was install but fastparquet wasn't
|
Owee, I'm MrMeeseeks, Look at me. There seem to be a conflict, please backport manually. Here are approximate instructions:
And apply the correct labels and milestones. Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon! Remember to remove the If these instructions are inaccurate, feel free to suggest an improvement. |
…ermediate versions
…#52211) * CI: Test pyarrow nightly instead of intermediate versions * Change format * Pin, remove hardcoded channel * Try pip * Fix some tests * Address more tests * Fix test condition * Fix another condidition * Cleanup name * Remove boto3
|
Do we have an issue to add back testing with multiple versions of pyarrow? I understand this PR reduced the CI load, but now we have several supported versions of pyarrow that we are not testing at all. And it actually seems that in practice, something else in the environment is causing a constraint on older pyarrow, so all our standard builds are using pyarrow 9.0.0. Which means we are not even testing the latest released pyarrow version at the moment? We have standard env files for python 3.9, 3.10 and 3.11 at the moment, we could easily test three different pyarrow versions in those? (xref #51308 (comment)) |
|
Testing our actions-311.yaml env file locally, and then trying to upgrade pyarrow causes a downgrade of pandas-gbq, so the bump of the minimum version of this might have caused to test with an older version of pyarrow. |
|
We don't have an open issue to discuss adding back testing with multiple version, but it would be good to have one to discuss. Yeah maybe we could have jobs that tests the required dependencies + pyarrow 8, 9, 10, ... |
Are there issues with pyarrow versions in the middle of our support range breaking? FWIW, for numpy we only test the min, the latest, and numpy dev, and it seems to work fine. Although the build matrix has been reduced by fusing jobs together, the individual jobs themselves take longer now (~1hr per job), so I'm still a little worried about the runners clogging up (We support a wide range of pyarrow rn, which is pyarrow 7-12 I think).
This seems like an issue. We could try adding a "clean" job without any dependencies, that would have pyarrow in it (this depends on PDEP-10 being accepted). In this case though, we are probably just pulling a very old version of pandas-gbq (mamba bug?). |
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.