[WIP]REGR: Fixed reading from public S3 buckets with credentials

[TomAugspurger]

Closes #34626

This works in 1.0.4 I think, so no whatsnew.

I'd like to wait for #34266 to get in first. I'll need to update things.

And in the future, I'd like to deprecate this behavior in favor of something like

pd.read_csv("s3://path/to/key", storage_options={"anon": True})

It should be easy to detect the cases when we need to do that. We just don't have the storage_options kwarg yet.

[WillAyd]

minor comment otherwise lgtm

[WillAyd]

Can you use the skip_if_no decorator from pandas/util/_test_decorators here instead?

[alimcmaster1]

Ahh just realised we are looking at the same thing @TomAugspurger . I think the case #34626 is trying to address is reading without AWS Credentials right? I've put a test together for this: #34877

[jorisvandenbossche]

The credentials @TomAugspurger is adding here are dummy (non-working) credentials to avoid a moto failure. But: since this is reading from an actual s3, and nothing is being mocked, moto should not be involved here?

[alimcmaster1]

The credentials @TomAugspurger is adding here are dummy (non-working) credentials to avoid a moto failure. But: since this is reading from an actual s3, and nothing is being mocked, moto should not be involved here?

Agree - think it should be as simple as: https://github.com/pandas-dev/pandas/pull/34877/files#diff-681243e864617f600083a270b329d17dR36 - unless i'm missing something?

[TomAugspurger]

I thought the issue was using credentials that are invalid for a public bucket. I don’t know what s3fs’ default behavior is when no credentials are found.

…

On Jun 20, 2020, at 04:29, Ali McMaster ***@***.***> wrote:  The credentials @TomAugspurger is adding here are dummy (non-working) credentials to avoid a moto failure. But: since this is reading from an actual s3, and nothing is being mocked, moto should not be involved here? Agree - think it should be as simple as: https://github.com/pandas-dev/pandas/pull/34877/files#diff-681243e864617f600083a270b329d17dR36 - unless i'm missing something? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[alimcmaster1]

@TomAugspurger - my understanding was:

import pandas as pd
res = pd.read_csv("s3://gdelt-open-data/events/1981.csv", nrows=3)

1. With AWS Credentials in my Env:
   pandas 1.0.3 & 1.0.4 -> Works

2. Without AWS Credentials in Env
   1.0.3 - Works
   1.0.4 - NoCredentialsError: Unable to locate credentials

I think (2) was the regression in #34626 - hence my PR here

But also think we should merge the case you are testing here ~ Dummy AWS Creds since maybe that tests (1) here?

[TomAugspurger]

Sounds good. Can you add the tests from my PR and I'll just close mine?

Regardless, I think we should wait till #34266 is in.

[alimcmaster1]

Sounds good. Can you add the tests from my PR and I'll just close mine?

Regardless, I think we should wait till #34266 is in.

Sure and apologies for the mix up. I've added this test to the PR here: #34877

Agree re waiting for fsspec.

[TomAugspurger]

Closing in favor of #34877.