Skip to content

CI: Test Pyarrow Less #46386

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from
Closed

CI: Test Pyarrow Less #46386

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
85e58bd
CI: Test Pyarrow Less
lithomas1 Mar 16, 2022
fda3a00
fixups
lithomas1 Mar 16, 2022
a5eac9b
comments
lithomas1 Mar 17, 2022
5cb5f9e
test latest pyarrow
lithomas1 Mar 17, 2022
73b7c30
update
lithomas1 Mar 23, 2022
7865aab
Merge branch 'main' into test-pyarrow-less
lithomas1 Mar 23, 2022
815127d
try something
lithomas1 Mar 26, 2022
98ce28f
Merge branch 'pandas-dev:main' into test-pyarrow-less
lithomas1 Mar 26, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 27 additions & 5 deletions .github/workflows/posix.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,6 @@ jobs:
matrix:
env_file: [actions-38.yaml, actions-39.yaml, actions-310.yaml]
pattern: ["not single_cpu", "single_cpu"]
# Don't test pyarrow v2/3: Causes timeouts in read_csv engine
# even if tests are skipped/xfailed
pyarrow_version: ["5", "7"]
include:
- env_file: actions-38-downstream_compat.yaml
pattern: "not slow and not network and not single_cpu"
Expand Down Expand Up @@ -62,8 +59,27 @@ jobs:
pandas_testing_mode: "deprecate"
test_args: "-W error"
name: "Numpy Dev"
# Test lowest & highest supported pyarrow
# Don't test pyarrow v2/3: Causes timeouts in read_csv engine
# even if tests are skipped/xfailed
- env_file: actions-38.yaml
pattern: "not single_cpu"
pyarrow_version: 5
name: "Pyarrow 5"
- env_file: actions-38.yaml
pattern: "single_cpu"
pyarrow_version: 5
name: "Pyarrow 5 (single)"
- env_file: actions-38.yaml
pattern: "not single_cpu"
pyarrow_version: latest
name: "Latest Pyarrow"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this eventually won't be the latest, similar better to name this "Pyarrow v7"

- env_file: actions-38.yaml
pattern: "single_cpu"
pyarrow_version: latest
name: "Latest Pyarrow (single)"
fail-fast: false
name: ${{ matrix.name || format('{0} pyarrow={1} {2}', matrix.env_file, matrix.pyarrow_version, matrix.pattern) }}
name: ${{ matrix.name || format('{0} {1}', matrix.env_file, matrix.pattern) }}
env:
ENV_FILE: ci/deps/${{ matrix.env_file }}
PATTERN: ${{ matrix.pattern }}
Expand Down Expand Up @@ -149,7 +165,13 @@ jobs:
if: ${{ env.IS_PYPY == 'false' }} # No pypy3.8 support

- name: Upgrade Arrow version
run: conda install -n pandas-dev -c conda-forge --no-update-deps pyarrow=${{ matrix.pyarrow_version }}
run: |
conda remove --force pyarrow
if [[ ${{matrix.pyarrow_version}} == 'latest' ]]; then
conda install -n pandas-dev -c conda-forge --no-update-deps pyarrow
else
conda install -n pandas-dev -c conda-forge --no-update-deps pyarrow=${{ matrix.pyarrow_version }}
fi
if: ${{ matrix.pyarrow_version }}

- name: Setup PyPy
Expand Down
6 changes: 3 additions & 3 deletions ci/deps/actions-310.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,16 +31,16 @@ dependencies:
- jinja2
- lxml
- matplotlib
# TODO: uncomment after numba supports py310
#- numba
- numba
- numexpr
- openpyxl
- odfpy
- pandas-gbq
- psycopg2
- pymysql
- pytables
- pyarrow
# Installed separately since mamba doesn't get latest version properly
#- pyarrow
- pyreadstat
- python-snappy
- pyxlsb
Expand Down
3 changes: 2 additions & 1 deletion ci/deps/actions-38.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,8 @@ dependencies:
- odfpy
- pandas-gbq
- psycopg2
- pyarrow
# Installed separately since mamba doesn't get latest version properly
#- pyarrow
- pymysql
- pyreadstat
- pytables
Expand Down
3 changes: 2 additions & 1 deletion ci/deps/actions-39.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,8 @@ dependencies:
- pandas-gbq
- psycopg2
- pymysql
- pyarrow
# Installed separately since mamba doesn't get latest version properly
#- pyarrow
- pyreadstat
- pytables
- python-snappy
Expand Down