str vs bytes confusion in ujson on Python 3

[rlamy]

Code Sample, a copy-pastable example if possible

import numpy as np
import pandas._libs.json as ujson

ujson.encode(np.array(1))

Problem description

On CPython 3.5, this raises TypeError, as expected, but the output contains garbage bytes:

Traceback (most recent call last):
  File "bug.py", line 4, in <module>
    ujson.encode(np.array(1))
TypeError: � (0d array) is not JSON serializable at the moment

On pypy3, it segfaults.

The problem comes from this code:

pandas/pandas/_libs/src/ujson/python/objToJSON.c

Lines 1925 to 1929 in 775099c

	tmpObj = PyObject_Repr(obj);
	PyErr_Format(PyExc_TypeError,
	"%s (0d array) is not JSON serializable at the moment",
	PyString_AS_STRING(tmpObj));
	Py_DECREF(tmpObj);

It calls PyString_AS_STRING (=PyBytes_AS_STRING on py3k) on the PyUnicodeObject returned by PyObject_Repr.

I haven't looked very carefully, but there seem to be a few more similar misuses of PyString_AS_STRING in the same file.

Expected Output

Traceback (most recent call last):
  File "bug.py", line 4, in <module>
    ujson.encode(np.array(1))
TypeError: array(1) (0d array) is not JSON serializable at the moment

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.3.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-135-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8
LOCALE: fr_FR.UTF-8

pandas: 0.21.1
pytest: 3.2.5
pip: 9.0.1
setuptools: 37.0.0
Cython: None
numpy: 1.13.3
scipy: None
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[sinhrks]

Thx for the report. PR is welcome.

[WillAyd]

I think this could just use the%R format substituting tempObj instead of the PyString_AS_STRING call:

https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_FromFormat

For anyone trying to improve C skills might be a good one to try out. Maybe @jbrockmendel