Please consider switch to standard json module

[zhihaoy]

Problem description

The ujson we currently using is not well maintained, no activity and no response since last two years:
ultrajson/ultrajson#291
and other projects are switching out of it:
fastavro/fastavro#150
It makes it hard for us to handle new feature requests:
#12213
and has bugs causing us unable to consume standard JSON files:
ultrajson/ultrajson#252

[jreback]

we use a vendored copy, so if you have a patch could be taken directly in pandas. the ujson is much more performant that the standard library and that is why it is used.

[zhihaoy]

There are benchmark showing that in large datasets, ujson doesn't show an advantage against standard library json. https://www.reddit.com/r/Python/comments/3mtswx/benchmark_of_pythons_alternative_json_libraries/ Moreover, we know as a fact that the json library in PyPy is the fastest (because it's optimized for JIT). Being said that, if someone really have concern about it, we can add a parameter to read_json families to let user chose which json-like library they want to use.

[TomAugspurger]

It's not just performance. The stdlib JSON module doesn't serialize things like NumPy arrays or scalars.

Does anyone know the status of Arrows's support for JSON (de)/serialization?

[jreback]

most folks do not use PyPy so not sure that actually matters

if you would like to add an engine kwarg to read_json would be ok

[zhihaoy]

It's not just performance. The stdlib JSON module doesn't serialize things like NumPy arrays or scalars.

The unreleased ujson upstream dropped this support since may 2016 (ultrajson/ultrajson@53f85b1), we still support it (buggyly ultrajson/ultrajson#221) by accident. If we want to keep this feature, we should do it properly by specializing on numpy.

[jreback]

I don't think arrow yet has support for JSON, cc @wesm @pitrou

[pitrou]

I'm not sure what you mean by support? Arrow C++ has some understanding of JSON, but it doesn't interoperate with Numpy or Pandas arrays.

[wesm]

@pitrou I think Jeff means support for JSON sufficient to power pandas.read_json. @bkietz is working on this in https://issues.apache.org/jira/browse/ARROW-694

I expect that we'll have support for reading and writing JSON sufficient to appease most pandas users on the timeline of Arrow 0.13 or 0.14, so most likely either by end of March or end of May

[jreback]

thanks @wesm yep that's what I mean. we have legacy c code in read_json and would be nice to remove it w/o sacrificing perf.

[jreback]

the c impl has recently been much refactored, closing this as no-action.