Support customised S3 servers endpoint URL

[xieqihui]

• closes #xxxx
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

Add support for customised S3 servers via checking the environment variable S3_ENDPOINT.

If set, the value of S3_ENDPOINT will be passed to the endpoint_url parameter for boto3.Session. It should be the complete URL to S3 service following the format: http(s)://{host}:{port}.

This feature is useful for companies who use their own S3 servers like MinIO, Ceph etc, as mentioned in issue #26195

[MarcoGorelli]

Hi xieqihui :) I believe there needs to be a type annotation, such as

        client_kwargs: Optional[Dict[str, str]] = {"endpoint_url": s3_endpoint}

[xieqihui]

Thanks for the reminder. I have added the type annotation and format the code.

[MarcoGorelli]

Part of the CI process checks that the code would be left unchanged by the black autoformatter.
Try running

black pandas/io/s3.py --diff

to see the changes (and leave out the --diff flag to apply them)

[TomAugspurger]

Is S3_ENDPOINT a common variable in the S3 ecosystem? Should s3fs just be using it as the default value for endpoint_url if it's not otherwise specified?

Ideally, pandas has as little s3-specific code as possible. We'd like to delegate it to other libraries.

[xieqihui]

Hi @TomAugspurger. In this commit, if S3_ENDPOINT is not specified, s3fs will be using the default value for endpoint_url, i.e., None, as documented in Boto3 client API. In this case, s3fs will connect to AWS S3 by default. Therefore, people need minimal effort to use s3 with pandas: for those who are using AWS S3, simply use it as it was before; while for people use their own s3 servers, define S3_ENDPOINT environment variable.

S3_ENDPOINT is a common variable to be set if people use their own S3 compatible object storage servers instead of AWS S3. This scenario is becoming more and more common as solutions like MinIO, Ceph gains more popularity. One example is that tensorflow supports S3 via S3_ENDPOINT environment variable. Pandas used to support this option by specifying AWS_S3_HOST before switching from boto to s3fs in this commit as discussed in #26195.

[jreback]

wouldn't this be better as a feature directly to s3fs?

[xieqihui]

Well, here comes an awkward situation: From the point view of s3fs and boto3, they already support customised endpoint by accepting parameters endpoint_url via client_kwargs, so there is no point to add environment variable support for this parameter there, as discussed on this boto3 issue.

The problem with Pandas now is that it doesn't provide a way to pass this parameter over to s3fs. I think adding support using this environment variable will benefit many people using their own s3 servers.

[TomAugspurger]

Why wouldn't using the environment variable as the default endpoint_url go in s3fs, to benefit even more people?

[TomAugspurger]

Agreed with Jeff. I would prefer to see this in s3fs.

[jreback]

closing as out of scope for pandas, encourage to submit to s3fs.

[martindurant]

Would appreciate is people on this thread would comment in fsspec/s3fs#273 as to how they would expect users to codify such arguments. I suppose environment variables is certainly an option, but passing arguments would be preferable. The linked issue talks about embedding these arguments in the URL itself.