Skip to content

na_rep Truncated With Int64 dtype in to_csv #25099

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mvinoba opened this issue Feb 2, 2019 · 1 comment · Fixed by #25103
Closed

na_rep Truncated With Int64 dtype in to_csv #25099

mvinoba opened this issue Feb 2, 2019 · 1 comment · Fixed by #25103
Labels
Bug ExtensionArray Extending pandas with custom dtypes or arrays. IO CSV read_csv, to_csv Output-Formatting __repr__ of pandas objects, to_string
Milestone

Comments

@mvinoba
Copy link
Contributor

mvinoba commented Feb 2, 2019

Code Sample

df = pd.DataFrame({"c": [float('nan')]*3})
df = df.astype("Int64")

print(df.to_csv(index=False, na_rep='mynull'))
c
myn
myn
myn

Problem description

I'd expect na_rep to be always respected, with all column types.

Expected Output

print(df.to_csv(index=False, na_rep='mynull'))
c
mynull
mynull
mynull

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.4.final.0
python-bits: 32
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: pt_BR.cp1252

pandas: 0.24.0
pytest: None
pip: 9.0.1
setuptools: 28.8.0
Cython: None
numpy: 1.16.1
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: None
patsy: None
dateutil: 2.7.5
pytz: 2018.9
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.2.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: 1.0.1
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

@WillAyd
Copy link
Member

WillAyd commented Feb 3, 2019

Thanks for the report - that does look rather strange. Investigation and PRs are always welcome!

@WillAyd WillAyd added Bug ExtensionArray Extending pandas with custom dtypes or arrays. IO CSV read_csv, to_csv labels Feb 3, 2019
@WillAyd WillAyd changed the title na_rep inherits maximum size of column's dtype when column is Int64 na_rep Truncated With Int64 dtype in to_csv Feb 3, 2019
@WillAyd WillAyd added this to the Contributions Welcome milestone Feb 3, 2019
kinow added a commit to kinow/pandas that referenced this issue Feb 3, 2019
kinow added a commit to kinow/pandas that referenced this issue Feb 3, 2019
kinow added a commit to kinow/pandas that referenced this issue Feb 3, 2019
kinow added a commit to kinow/pandas that referenced this issue Feb 3, 2019
kinow added a commit to kinow/pandas that referenced this issue Feb 3, 2019
kinow added a commit to kinow/pandas that referenced this issue Feb 3, 2019
kinow added a commit to kinow/pandas that referenced this issue Feb 3, 2019
kinow added a commit to kinow/pandas that referenced this issue Feb 4, 2019
@jreback jreback modified the milestones: Contributions Welcome, 0.25.0 Feb 6, 2019
kinow added a commit to kinow/pandas that referenced this issue Mar 20, 2019
kinow added a commit to kinow/pandas that referenced this issue Mar 20, 2019
kinow added a commit to kinow/pandas that referenced this issue Apr 10, 2019
kinow added a commit to kinow/pandas that referenced this issue Apr 11, 2019
kinow added a commit to kinow/pandas that referenced this issue Apr 21, 2019
kinow added a commit to kinow/pandas that referenced this issue Jun 1, 2019
@simonjayhawkins simonjayhawkins added the Output-Formatting __repr__ of pandas objects, to_string label Jun 14, 2019
@jreback jreback modified the milestones: 0.25.0, Contributions Welcome Jun 28, 2019
kinow added a commit to kinow/pandas that referenced this issue Jul 10, 2019
kinow added a commit to kinow/pandas that referenced this issue Sep 6, 2019
kinow added a commit to kinow/pandas that referenced this issue Sep 9, 2019
kinow added a commit to kinow/pandas that referenced this issue Sep 9, 2019
kinow added a commit to kinow/pandas that referenced this issue Sep 9, 2019
@TomAugspurger TomAugspurger modified the milestones: Contributions Welcome, 1.0 Sep 10, 2019
TomAugspurger pushed a commit that referenced this issue Sep 10, 2019
…uncation (#25103)

* Fix #25099 set na_rep values before converting to string to prevent data truncation
proost pushed a commit to proost/pandas that referenced this issue Dec 19, 2019
…uncation (pandas-dev#25103)

* Fix pandas-dev#25099 set na_rep values before converting to string to prevent data truncation
proost pushed a commit to proost/pandas that referenced this issue Dec 19, 2019
…uncation (pandas-dev#25103)

* Fix pandas-dev#25099 set na_rep values before converting to string to prevent data truncation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug ExtensionArray Extending pandas with custom dtypes or arrays. IO CSV read_csv, to_csv Output-Formatting __repr__ of pandas objects, to_string
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants