OverflowError on using to_json to serialize NaN value with type Decimal.

[ash-ishh]

 import pandas as pd
 from decimal import Decimal

 def handle_nan(value):
     if isinstance(value, Decimal):
         return float(value)
     return value

 temp_dict = {'a': [Decimal(1), Decimal('NaN')]}
 df = pd.DataFrame(temp_dict)
 df_json = df.to_json(default_handler=handle_nan)

Error

OverflowError: Invalid Nan value when encoding double.

Problem Description

Default handler not being called while getting error in serialization.

Expected Output

handle_nan should be called to return float NaN which is serializable.

INSTALLED VERSIONS commit : None python : 3.6.8.final.0 python-bits : 64 OS : Linux OS-release : 4.4.187-1.el7.elrepo.x86_64 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 0.25.1
numpy : 1.17.0
pytz : 2019.2
dateutil : 2.8.0
pip : 18.1
setuptools : 40.6.2
Cython : 0.28.2
pytest : 5.1.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 1.0.4
lxml.etree : 4.4.1
html5lib : 1.0.1
pymysql : 0.9.3
psycopg2 : 2.8.3 (dt dec pq3 ext lo64)
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : 4.8.0
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : 4.4.1
matplotlib : None
numexpr : None
odfpy : None
openpyxl : 2.6.2
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : 1.1.0
sqlalchemy : 1.3.3
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : 1.0.4

[bluppfisk]

+1 We have a use case in which we perform calculations with Decimals in pandas. There is inconsistency between DataFrame.to_json() handling of numpy.nan and decimal.Decimal('NaN'). The latter throws the OverflowError mentioned above. The first one is simply converted to null.

Is this intended behaviour?