to_csv writes wrong with NaN value

[jackasser]

to_csv with Nan value at top row, unexpected "" in the csv file

import pandas as pd
df = pd.DataFrame([None,1,2])
df.to_csv("df.csv",header=None,index=None,encoding ='utf-8')

# df.csv
#""
#1.0
#2.0

# I want to make df.csv
#
#1.0
#2.0

Versions:

pandas.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.2.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 61 Stepping 4, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.20.3
pytest: None
pip: 9.0.1
setuptools: 36.4.0
Cython: None
numpy: 1.13.1
scipy: 0.19.1
xarray: None
IPython: 6.1.0
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.9999999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None
​

[jreback]

this does look like a bug.

@gfyoung ?

[gfyoung]

Agreed. I can also replicate this on 0.21.0 (@jackasser : I noticed that you were on 0.20.3). Investigation and PR are welcome!

[Licht-T]

I am working on this. This seems not the encoding issue.

df.to_csv("df.csv",header=None,index=None)

[pandas] cat df.csv                                                                                                       14:54:23  ☁  add-tuple-list-factorize-test ☂ ✭
""
1.0
2.0

[Licht-T]

Also, this returns the right result.

df = pd.DataFrame([1,None,1,2])

[Licht-T]

@gfyoung Seems that the bug or the default spec. in csv library.

Case 1

Input:

import csv
fp = open('test.csv', 'w')
w = csv.writer(fp, dialect=csv.excel)
w.writerow(['1'])
w.writerow([''])
fp.close()

Output:

1
　

Case 2

Input:

import csv
fp = open('test.csv', 'w')
w = csv.writer(fp, dialect=csv.excel)
w.writerow([''])
w.writerow(['1'])
fp.close()

Output:

""
1

[Licht-T]

But this works.

import csv
fp = open('test.csv', 'w')
w = csv.writer(fp, dialect=None)
w.writerow(['', '1'])
w.writerow(['3', '2'])
fp.close()

,1
3,2

[Licht-T]

The csv library in CPython says the single empty field is quoted.
https://github.com/python/cpython/blob/0b3ec192259a65971001ce8f0de85a9c1e71d9c7/Modules/_csv.c#L1244

I don't know why this is needed.

[gfyoung]

@Licht-T : Ah! That's very good to know. Okay, this means that this issue is out of the control of the pandas library since we do wrap Python csv library when doing this.

@jackasser : Looks like you may have hit upon a point of contention in Python's CSV library. I would raise this issue in their library by submitting an issue on their python.org website.

[gfyoung]

Closing because this is out of pandas control. @Licht-T : Thanks for the investigation!