BUG:dtype=object columns with NaNs coerce integers to floats

[ashtonteng]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

df = pd.DataFrame({'name': [np.nan, "hi"],
                   'mask': [True, 4],
                   'weapon': [5, np.nan]}).astype(object)
print(df) 
"""
name  mask weapon
0  NaN  True      5
1   hi     4    NaN
""" # everything prints out correctly
print(df.iloc[0].values) # [nan True 5.0] # during indexing, 5 gets coerced to 5.0
# ultimately I want to be able to do df.to_csv(), and get unmodified data

Problem description

I have a dataframe of mixed types, which I cast to dtype=object, and NaNs in certain grids. Printing out the dataframe works fine, but when I try indexing into specific rows, or exporting to_csv, the values in the columns with NaNs are quietly modified - integers become floats. I believe this is a bug because I would expect to_csv to produce a csv file that is similar to printing the dataframe, without data types being changed. I cannot cast the dataframe to type Int64, since I have other types in the dataframe as well.

Expected Output

Instead of [nan True 5.0], I should get [nan True 5]

Output of pd.show_versions()

INSTALLED VERSIONS

commit : d9fff27
python : 3.7.8.final.0
python-bits : 64
OS : Darwin
OS-release : 19.6.0
Version : Darwin Kernel Version 19.6.0: Sun Jul 5 00:43:10 PDT 2020; root:xnu-6153.141.1~9/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.1.0
numpy : 1.19.1
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.1
setuptools : 49.2.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.16.1
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.0
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.5.2
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[dsaxton]

That's interesting. I don't know what's happening under the hood there (maybe someone else can chime in) but notice that your original value is a float and not an integer. It's more surprising to me that it "looks like" an integer when cast to object, not that the eventual numpy representation is float (which I think is expected). Here's a slightly smaller example by the way that shows the same thing:

pd.Series([1.0]).astype(object)
# 0    1
# dtype: object
pd.Series([1.0]).astype(object).values
# array([1.0], dtype=object)

If it needs to be an integer in the end you should use a nullable integer dtype for that column in the original DataFrame (you can cast specific columns, no need to try to convert the whole thing to Int64).

[simonjayhawkins]

I think that there maybe a bug here with the float formatting in the object column.

to answer the OP.

@dsaxton is correct that values used in the DataFrame constructor will be floats. This can be avoided by passing dtype to the constructor (or creating Series with object dtype upfront and creating DataFrame from Series instead of dict)

>>> pd.__version__
'1.2.0.dev0+46.g9c566a635'
>>>
>>> df = pd.DataFrame(
...     {"name": [np.nan, "hi"], "mask": [True, 4], "weapon": [5, np.nan]}, dtype=object
... )
>>>
>>> print(df)
  name  mask weapon
0  NaN  True      5
1   hi     4    NaN
>>>
>>> print(df.iloc[0].values)
[nan True 5]
>>>

using .astype is a separate operation after the DataFrame is constructed and the values coerced to floats to accommodate the missing values

>>> df = pd.DataFrame({"name": [np.nan, "hi"], "mask": [True, 4], "weapon": [5, np.nan]})
>>> df.dtypes
name       object
mask       object
weapon    float64
dtype: object
>>>

i'll open a separate issue for the formatting issue and close this as answered.

[simonjayhawkins]

i'll open a separate issue for the formatting issue and close this as answered.

#35603