Add ability to specify AWS s3 host through AWS_S3_HOST environment variable #12198
Conversation
|
Can this be configured through boto? I think |
|
Unfortunately, it can't. You can't add it as an option to the .boto configuration either. Other packages have implemented a similar work-around. |
| @@ -274,14 +274,15 @@ def get_filepath_or_buffer(filepath_or_buffer, encoding=None, | |||
| import boto | |||
| except: | |||
| raise ImportError("boto is required to handle s3 files") | |||
| # Assuming AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY | |||
| # Assuming AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_S3_HOST | |||
There was a problem hiding this comment.
can we document this in some doc-strings and/or a separate section in the docs? (e.g. it may be too long for the actual doc-strings, so might be better to make a section on how to do S3 access and just point to it).
There was a problem hiding this comment.
I agree, it would be nice to have S3 documented in a dedicated section. Can that be its own GH issue so that this feature/bug can be pushed through faster?
I am not very familiar with the Pandas doc-string/documentation strategy for the read_* functions, so this may be something for a core contributor to work on.
There was a problem hiding this comment.
I would like to make a little section at least to document this new env variable, so can you add a simple section to the docs (which can be built out later).
The AWS s3 host used by read_csv currently defaults to s3.amazonaws.com and there is currently no way to change this. This pull requests adds functionality where the code checks the AWS_S3_HOST environment variable for a user-specified host and defaults back to the current s3.amazonaws.com host.
This is a simple code change, but it will allow users from around the globe to use read_csv to directly read from files in s3 when the s3 bucket is not located in the US East region.