Skip to content

to_csv swallows exception when writing to S3 #30732

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vfilimonov opened this issue Jan 6, 2020 · 3 comments
Closed

to_csv swallows exception when writing to S3 #30732

vfilimonov opened this issue Jan 6, 2020 · 3 comments
Assignees
Labels
Bug IO CSV read_csv, to_csv IO Network Local or Cloud (AWS, GCS, etc.) IO Issues

Comments

@vfilimonov
Copy link
Contributor

vfilimonov commented Jan 6, 2020

I'm not sure if this issue belongs to pandas or s3fs.

When writing to non-existing bucket or bucket without proper permissions no exception is raised. E.g. the following code will be executed normally:

df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
df.to_csv('s3://very.weird.and.certainly.nonexistent.bucket/data.csv')  # No exception

In contrast, when writing to a local file without proper permissions results in an exception as it should be:

dff.to_csv('/data.csv')  # PermissionError: [Errno 13] Permission denied: '/data.csv'

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.1.final.0
python-bits : 64
OS : Darwin
OS-release : 18.7.0
machine : x86_64
processor : i386
byteorder : little
LC_ALL : en_US.UTF-8
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 0.25.3
numpy : 1.17.2
pytz : 2019.3
dateutil : 2.8.1
pip : 19.3.1
setuptools : 42.0.1.post20191125
Cython : None
pytest : 5.3.0
hypothesis : None
sphinx : 2.2.0
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.4.1
html5lib : None
pymysql : 0.9.3
psycopg2 : None
jinja2 : 2.10.1
IPython : 7.8.0
pandas_datareader: None
bs4 : 4.6.3
bottleneck : None
fastparquet : 0.3.2
gcsfs : None
lxml.etree : 4.4.1
matplotlib : 3.0.1
numexpr : 2.7.0
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 0.12.1
pytables : None
s3fs : 0.4.0
scipy : 1.3.1
sqlalchemy : 1.3.8
tables : 3.4.4
xarray : None
xlrd : 1.1.0
xlwt : None
xlsxwriter : None

@gfyoung gfyoung added the IO Data IO issues that don't fit into a more specific label label Jan 7, 2020
@gfyoung
Copy link
Member

gfyoung commented Jan 7, 2020

I would look into seeing where we call s3fs to do the actual writing. I would be surprised if it was s3fs thing that it wouldn't tell us if a bucket didn't exist.

@jbrockmendel jbrockmendel added the IO CSV read_csv, to_csv label Feb 25, 2020
@mroeschke mroeschke added Bug IO Network Local or Cloud (AWS, GCS, etc.) IO Issues and removed IO Data IO issues that don't fit into a more specific label labels May 2, 2020
@alimcmaster1
Copy link
Member

Take

@alimcmaster1
Copy link
Member

Duplicate of #32486 I think. Should now be fixed on master by #33645

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug IO CSV read_csv, to_csv IO Network Local or Cloud (AWS, GCS, etc.) IO Issues
Projects
None yet
Development

No branches or pull requests

5 participants