Skip to content

Not able to save (to_csv) large file #21643

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
VelizarVESSELINOV opened this issue Jun 26, 2018 · 3 comments
Closed

Not able to save (to_csv) large file #21643

VelizarVESSELINOV opened this issue Jun 26, 2018 · 3 comments

Comments

@VelizarVESSELINOV
Copy link

Code Sample, a copy-pastable example if possible

"""Pandas bug."""
from numpy.random import choice, seed
from pandas import DataFrame
from pandas.util.testing import rands_array

seed(1234)
N = 5 * 10 ** 6

DTF = DataFrame({'Col{}'.format(k): choice(rands_array(10, 750), N) for k in range(65)})

print(DTF)

DTF.to_csv('File{}.csv'.format(N)

Problem description

to_csv exception, eventually linked to latest version regression #21471 issue with to_csv with compression

Expected Output

Expected no exception

  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/core/frame.py", line 1745, in to_csv
    formatter.save()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/io/formats/csvs.py", line 166, in save
    f.write(buf)
OSError: [Errno 22] Invalid argument

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Darwin
OS-release: 17.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.1
pytest: None
pip: 10.0.1
setuptools: 39.0.1
Cython: None
numpy: 1.14.5
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: 1.1.0
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

@jreback
Copy link
Contributor

jreback commented Jun 26, 2018

as noted this is. duplicate

@jreback jreback closed this as completed Jun 26, 2018
@VelizarVESSELINOV
Copy link
Author

Hopefully, a new version with a fix will come soon, the last 2 versions (42 days) are "unusable" due to 2 blocking regressions (groupby.ffill() and now to_csv not working).
I hope Pandas basic functionalities code coverage will increase with all the examples that people are submitting to show you the different bugs.

@jreback
Copy link
Contributor

jreback commented Jun 26, 2018

you are welcome to submit fixes if you think things are lacking

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants