Skip to content

Support customised S3 servers endpoint URL #29050

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

Support customised S3 servers endpoint URL #29050

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f9da50a
Support customised S3 servers endpoint URL
Oct 17, 2019
febbdfe
add type annotation and format s3.py
Oct 18, 2019
26885ed
Explain s3_endpoint
Oct 18, 2019
690752b
fix type annotation in s3.py
Oct 18, 2019
1f24b98
isort s3.py
Oct 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions pandas/io/s3.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
""" s3 support for remote file interactivity """
import os
from typing import IO, Any, Optional, Tuple
from urllib.parse import urlparse as parse_url

Expand All @@ -25,7 +26,15 @@ def get_file_and_filesystem(
if mode is None:
mode = "rb"

fs = s3fs.S3FileSystem(anon=False)
# Support customised S3 servers endpoint URL via environment variable
# The S3_ENDPOINT should be the complete URL to S3 service following
# the format: http(s)://{host}:{port}
s3_endpoint = os.environ.get('S3_ENDPOINT')
Copy link
Member

@MarcoGorelli MarcoGorelli Oct 17, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Part of the CI process checks that the code would be left unchanged by the black autoformatter.
Try running

black pandas/io/s3.py --diff

to see the changes (and leave out the --diff flag to apply them)

if s3_endpoint:
client_kwargs = {'endpoint_url': s3_endpoint}
Copy link
Member

@MarcoGorelli MarcoGorelli Oct 17, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi xieqihui :) I believe there needs to be a type annotation, such as

        client_kwargs: Optional[Dict[str, str]] = {"endpoint_url": s3_endpoint}

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the reminder. I have added the type annotation and format the code.

else:
client_kwargs = None
fs = s3fs.S3FileSystem(anon=False, client_kwargs=client_kwargs)
try:
file = fs.open(_strip_schema(filepath_or_buffer), mode)
except (FileNotFoundError, NoCredentialsError):
Expand All @@ -35,7 +44,7 @@ def get_file_and_filesystem(
# aren't valid for that bucket.
# A NoCredentialsError is raised if you don't have creds
# for that bucket.
fs = s3fs.S3FileSystem(anon=True)
fs = s3fs.S3FileSystem(anon=True, client_kwargs=client_kwargs)
file = fs.open(_strip_schema(filepath_or_buffer), mode)
return file, fs

Expand Down