BUG: Unexpected output of to_json for complex data

[karalekas]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

import pandas as pd
s = pd.Series({0: 0+2j, 1: 3+4j, 2: 5+6j})
s.to_json()

Problem description

Calling to_json() on a DataFrame or Series with complex data gives unexpected results. Running the above Code Sample gives the following output:

'{"0":{"imag":2.0},"1":{"imag":4.0},"2":{"imag":6.0}}'

This is a problem both because (1) the output is not terribly useful (the real part is just dropped), and (2) according to the docs this should raise an error (see the following snippet copied from the linked docs page):

>>> DataFrame([1.0, 2.0, complex(1.0, 2.0)]).to_json()  # raises
RuntimeError: Unhandled numpy dtype 15

Expected Output

According to the docs, this should at least raise a RuntimeError. However, it would be nice for this to actually be supported.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 0158382
python : 3.8.5.final.0
python-bits : 64
OS : Darwin
OS-release : 20.1.0
Version : Darwin Kernel Version 20.1.0: Sat Oct 31 00:07:11 PDT 2020; root:xnu-7195.50.7~2/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.3.0.dev0+1441.g0158382ac8
numpy : 1.20.2
pytz : 2021.1
dateutil : 2.8.1
pip : 20.1.1
setuptools : 47.1.0
Cython : 0.29.23
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 7.22.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[pablodz]

Same with dataframes

import pandas as pd
df  = pd.DataFrame(np.random.randn(2, 3) + 1j*np.random.randn(2, 3))
json=df.to_json()
print(json)

prints

'{"0":{"0":{"imag":0.0286552288},"1":{"imag":-1.095500714}},"1":{"0":{"imag":-1.823351728},"1":{"imag":-0.732113612}},"2":{"0":{"imag":0.5109315559},"1":{"imag":0.8221189199}}}'

# From a dataframe of
                    0                   1                   2
0  0.290380+0.028655j -0.171614-1.823352j  0.910000+0.510932j
1 -1.236152-1.095501j  1.565766-0.732114j  0.324258+0.822119j

[pablodz]

By default of to_json() is

pandas/pandas/io/json/_json.py

Lines 78 to 139 in 070341c

	def to_json(
	path_or_buf,
	obj: NDFrame,
	orient: Optional[str] = None,
	date_format: str = "epoch",
	double_precision: int = 10,
	force_ascii: bool = True,
	date_unit: str = "ms",
	default_handler: Optional[Callable[[Any], JSONSerializable]] = None,
	lines: bool = False,
	compression: CompressionOptions = "infer",
	index: bool = True,
	indent: int = 0,
	storage_options: StorageOptions = None,
	):
	if not index and orient not in ["split", "table"]:
	raise ValueError(
	"'index=False' is only valid when 'orient' is 'split' or 'table'"
	)
	if lines and orient != "records":
	raise ValueError("'lines' keyword only valid when 'orient' is records")
	if orient == "table" and isinstance(obj, Series):
	obj = obj.to_frame(name=obj.name or "values")
	writer: Type[Writer]
	if orient == "table" and isinstance(obj, DataFrame):
	writer = JSONTableWriter
	elif isinstance(obj, Series):
	writer = SeriesWriter
	elif isinstance(obj, DataFrame):
	writer = FrameWriter
	else:
	raise NotImplementedError("'obj' should be a Series or a DataFrame")
	s = writer(
	obj,
	orient=orient,
	date_format=date_format,
	double_precision=double_precision,
	ensure_ascii=force_ascii,
	date_unit=date_unit,
	default_handler=default_handler,
	index=index,
	indent=indent,
	).write()
	if lines:
	s = convert_to_line_delimits(s)
	if path_or_buf is not None:
	# apply compression and byte/text conversion
	with get_handle(
	path_or_buf, "w", compression=compression, storage_options=storage_options
	) as handles:
	handles.handle.write(s)
	else:
	return s

I can't find the RuntimeError: Unhandled numpy dtype 15 expected output

[mroeschke]

I'm getting a more useful result on master now that could use tests to nail this behavior down

In [13]: import pandas as pd
    ...: s = pd.Series({0: 0+2j, 1: 3+4j, 2: 5+6j})
    ...: s.to_json()
Out[13]: '{"0":{"imag":2.0,"real":0.0},"1":{"imag":4.0,"real":3.0},"2":{"imag":6.0,"real":5.0}}'

[lorenzophys]

@mroeschke I would like to write the tests for this behavior. Being my first issue with this codebase, tests related to this issue will go in pandas/tests/io/json/test_pandas.py? Or there's a more appropriate place?

[mroeschke]

@lorenzophys yes that sounds like a good location