CI testing for s3 reads from public buckets on 1.0.x - DO NOT MERGE

[simonjayhawkins]

xref #34626

[simonjayhawkins]

@jorisvandenbossche this works for me locally. If you can have a look at the failures in Travis and let me know if we're confident that #34626 is fixed on 1.0.x

[jorisvandenbossche]

Does the test run on one of the azure builds as well?

[jorisvandenbossche]

cc @TomAugspurger do you have any experience with this? On travis, the S3 public bucket reading is failing with boto / permission errors.

I checked out the 1.0.x branch locally, and can confirm it is working locally.

[TomAugspurger]

Not sure. I notice that the two jobs with failing moto / boto issues have moto==1.3.7. https://travis-ci.org/github/pandas-dev/pandas/jobs/698478225#L1128 has 1.3.14. Perhaps we failed to backport a dependency update?

[jorisvandenbossche]

But moto shouldn't be used for that test? (unless that is done automatically?)

[TomAugspurger]

Ah I missed that this was for the public bucket. You're correct, I think moto shouldn't be involed here..

[TomAugspurger]

When you're testing locally, do you run the whole file or just the single test? s3fs does cache things, including the S3FSFileSystem constructor, which might cause this.

[simonjayhawkins]

Does the test run on one of the azure builds as well?

i'll create a failing test to make sure

[jorisvandenbossche]

When you're testing locally, do you run the whole file or just the single test? s3fs does cache things, including the S3FSFileSystem constructor, which might cause this.

I just ran pd.read_csv("s3://nyc-tlc/misc/taxi _zone_lookup.csv") in a console

[jorisvandenbossche]

But running it with pytest pandas/tests/io/test_s3.py also works

[simonjayhawkins]

if we get the tests passing, does pandas have an amazon account where we can place a test file in a public bucket?

[simonjayhawkins]

hmm travis didn't run.

Does the test run on one of the azure builds as well?

on 3 azure builds, will revert the fail shortly

[jorisvandenbossche]

OK, if it thus was passing on azure, I think we can be confident enough for the 1.0.5 release that this is properly working. And can then figure out the travis failure later when we fix this in master

[simonjayhawkins]

OK, if it thus was passing on azure, I think we can be confident enough for the 1.0.5 release that this is properly working. And can then figure out the travis failure later when we fix this in master

OK closing

[jorisvandenbossche]

We could also already merge it (but then in the master branch), with an xfail, if you want

[simonjayhawkins]

i'm not sure about the ethics of using someones public bucket for testing

[alimcmaster1]

Ref my comment here: #34626 (comment)

I couldn’t get a test for public buckets working using moto.

I propose we use an Amazon Open Data public s3 bucket to test https://aws.amazon.com/opendata/public-datasets/

We can use the tm.network decorator? Would be helpful to catch future regressions - especially since we are moving to fsspec. Thoughts @jorisvandenbossche @simonjayhawkins

I can open a PR.

[jorisvandenbossche]

Using an existing public s3 bucket seems fine for this (we should just find a small dataset).
And indeed, we have network tests, so we can just test this for real, without moto, with the network decorator. Thanks @alimcmaster1 !