TST: Fix test_parquet failures for pyarrow 1.0

[alimcmaster1]

• closes BUG: failing test for pyarrow/s3fs on windows #35791
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff

cc @martindurant -> looks like this is a pyarrow 1.0.0 compat issue (read_table uses the new API) - https://arrow.apache.org/docs/python/generated/pyarrow.parquet.read_table.html

I noticed the partition cols are casted from int64 -> int32 is that expected pyarrow behaviour? From the write_table docs looking at version 1.0/2.0 it suggests it is https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html#pyarrow.parquet.write_table

[martindurant]

May be best asking @jorisvandenbossche about intended behaviour

[jorisvandenbossche]

@alimcmaster1 thanks for looking into this!

Yes, so with pyarrow 1.0 and using the new datasets implementation, we kept the default for category type if your partition field is string, but not for integers (and indeed also the default now is int32, not int64)

So I suppose out other roundtrip tests are using string partition fields (since this test is the only one that is failing)

[jorisvandenbossche]

So I suppose out other roundtrip tests are using string partition fields (since this test is the only one that is failing)

Actually it doesn't see we have other full roundtrip tests that use partitioning .. (only tests checking that it is written correctly)

[jorisvandenbossche]

so only for integer columns the behaviour changed, for string columns it is still using category type

[alimcmaster1]

@alimcmaster1 thanks for looking into this!

Yes, so with pyarrow 1.0 and using the new datasets implementation, we kept the default for category type if your partition field is string, but not for integers (and indeed also the default now is int32, not int64)

So I suppose out other roundtrip tests are using string partition fields (since this test is the only one that is failing)

Gotcha makes sense - thanks for the info!

[jreback]

@alimcmaster1 can you rebase. I marked for 1.1.2 i think is ok (not sure if it is failing on that).

[jreback]

@jorisvandenbossche lgtm.

[jorisvandenbossche]

@meeseeksdev backport to 1.1.x

[lumberbot-app]

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

$ git checkout 1.1.x
$ git pull

2. Cherry pick the first parent branch of the this PR on top of the older branch:

$ git cherry-pick -m1 d3d74c590e2578988a2be48d786ddafa89f91454

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

$ git commit -am 'Backport PR #35814: TST: Fix test_parquet failures for pyarrow 1.0'

4. Push to a named branch :

git push YOURFORK 1.1.x:auto-backport-of-pr-35814-on-1.1.x

5. Create a PR against branch 1.1.x, I would have named this PR:

"Backport PR #35814 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to suggest an improvement.