BUG: Series.to_json returning a native python float from a column of type int64

[mbiokyle29]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

import pandas
example_df = pandas.DataFrame({'a': [1], 'b': [1.0]})

# this passes (DataFrame.to_dict is called)
assert type(example_df.to_dict('records')[0]['a']) is int

# this passes (Series.to_dict is called but on a Series with no floats)
assert type(example_df[['a']].iloc[0].to_dict()['a']) is int

# this fails
assert type(example_df.iloc[0].to_dict()['a']) is int

# this fails
assert type(example_df.apply(pandas.Series.to_dict, axis=1).iloc[0]['a']) is int

Problem description

I have found another issue which involves type conversion / handling with the DataFrame/Series to_dict function. There are a number of other issues in this area that I found, but I think this one is unique.

Related issues:

• to_dict doesn't convert np.int64 to python integers (second try) #24908: this is an issue where the output is not being converted to the native python data type
• Series iteration and to_dict methods *sometimes* return underlying storage type vs. Python object #25969: again, this is "numpy -> native" type conversion not happening
• to_dict() on a boolean series sometimes returns numpy types instead of Python types #27616: numpy.boolean being returned

In most/all the cases I could find, the issue was that to_dict was returning the underlying numpy type and not converting to the native. In this case, I am getting a native python type, just the wrong one. It only happens when both are true:

• Series.to_dict is called
• The Series in question contains an int64 and a float64 column

Expected Output

I would expect all 3 assert statements to pass, which is to say that I should get data back with int type for the 'a' column. Visually speaking:

In [7]: example_df.to_dict('records')[0]
Out[7]: {'a': 1, 'b': 1.0}

In [8]: example_df.iloc[0].to_dict()
Out[8]: {'a': 1.0, 'b': 1.0}. # a should not be a float here

In [11]: type(example_df.iloc[0].to_dict()['a'])
Out[11]: float  # this should be int

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 9d598a5
python : 3.9.1.final.0
python-bits : 64
OS : Darwin
OS-release : 20.1.0
Version : Darwin Kernel Version 20.1.0: Sat Oct 31 00:07:11 PDT 2020; root:xnu-7195.50.7~2/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.2.1
numpy : 1.19.5
pytz : 2020.5
dateutil : 2.8.1
pip : 20.3.3
setuptools : 51.1.1
Cython : None
pytest : 6.2.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 7.19.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.3
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[phofl]

Hi, thanks for your report.

I don't think that these are bugs.

Your first failure:

example_df.iloc[0]

returns the first row as a Series, meaning that we have to find a common dtype -> float. Afterwards we do not cast back of course when you cast to a dict and select a.

Second failure:

example_df.apply(pd.Series.to_dict, axis=1)

Similar here. Row-wise conversion to Series -> common dtype -> conversion to dict but it is already a float. Your iloc selection afterwards does not make a difference.

Edit: The others work because we do not operate row-wise.

[mbiokyle29]

Hi, @phofl

Thanks for the quick response. This is clearly an issue of me not understanding how Series works! Sorry to waste your time.

[phofl]

No problem.

If you wanna keep the dtype, select the column first and then the row, this should help a bit

[mbiokyle29]

My initial issue was trying to collect 1 or more columns from a dataframe, and collapse them into a single column (storing the json dump of the Series. I think the best way to do it is as follows:

df[['a', 'b']].astype(object).apply(
    pandas.Series.to_json,
    axis=1,
 )

The astype(object) seems to be key here. I am guessing that it sets the dtype of each row wise Series to object and thus no type info is lost in the "common" translation.