read_json support for orient="table" #19039

WillAyd · 2018-01-02T21:13:19Z

closes read_json can't import own exported data with orient=table #18912 and
closes Roundtrip orient for JSON #9146
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

This is a starting point for read_json supporting orient='table' for basic types. I'm explicitly raising in the instances that don't work, namely timezone aware datetimes andTimedelta. I marked floats with no decimal values as an xfail (these are being converted to int64 by the _try_convert_data method).

Timedelta would be best fixed in a separate change providing a constructor equivalent to the isoformat method (see #19040).

The new tests could also all be parametrized, but I've kept as is before going too far with this change in case of feedback

gfyoung · 2018-01-02T22:04:59Z

pandas/io/json/table_schema.py

@@ -103,6 +110,28 @@ def make_field(arr, dtype=None):
    return field


+def revert_field(field):


Needs a docstring

gfyoung · 2018-01-02T22:05:18Z

pandas/io/json/table_schema.py

+def parse_table_schema(json, precise_float):
+    """
+    Builds a DataFrame from a given schema
+    """


Expand this docstring to discuss parameters and return value.

Also, document the NotImplementedError cases.

gfyoung · 2018-01-02T22:06:23Z

pandas/tests/io/json/test_json_table_schema.py

+
+    @pytest.mark.parametrize("index_names", [[None, None], ['foo', 'bar']])
+    def test_multiindex(self, index_names):
+        # GH 18912


Why are you adding the issue reference here only?

Only because the issue was where I pulled the code sample from. I was hesitant to include in the first place because the issue ends up touching on something larger than originally intended, namely that there is no read_json support for orient="table". Would you rather remove or do you think it's worth moving that reference to all of the test cases?

As you said, given that your PR adds an entire new class of tests, I actually think what you've done is fine (since the issue directly points to the test). I was looking for some clarification.

gfyoung · 2018-01-02T22:17:18Z

pandas/io/json/json.py

@@ -839,6 +839,9 @@ def _parse_no_numpy(self):
        elif orient == "index":
            self.obj = DataFrame(
                loads(json, precise_float=self.precise_float), dtype=None).T
+        elif orient == 'table':


Add to docstring that supported was added for this option in 0.23.0.

codecov · 2018-01-03T06:42:45Z

Codecov Report

Merging #19039 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #19039      +/-   ##
==========================================
+ Coverage   91.53%   91.54%   +<.01%     
==========================================
  Files         148      148              
  Lines       48688    48729      +41     
==========================================
+ Hits        44566    44607      +41     
  Misses       4122     4122

Flag	Coverage Δ
#multiple	`89.91% <100%> (ø)`	⬆️
#single	`41.6% <17.02%> (-0.03%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/json/table_schema.py	`98.19% <100%> (+0.97%)`	⬆️
pandas/io/json/json.py	`92.6% <100%> (+0.03%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f2d8db1...36cab57. Read the comment docs.

jreback · 2018-01-03T11:28:03Z

pandas/io/json/table_schema.py

@@ -103,6 +108,40 @@ def make_field(arr, dtype=None):
    return field


+def revert_field(field):
+    '''


use triple-double quotes rather than single

jreback · 2018-01-03T11:29:23Z

pandas/io/json/table_schema.py

@@ -103,6 +108,40 @@ def make_field(arr, dtype=None):
    return field


+def revert_field(field):


can we rename make_field and this to something more descriptive. make
convert_json_typename_to_pandas (and the reverse)

jreback · 2018-01-03T11:30:00Z

pandas/io/json/table_schema.py

+    Returns
+    -------
+    dtype
+    '''


can you add an example of tz and categorical here

jreback · 2018-01-03T11:31:15Z

pandas/tests/io/json/test_json_table_schema.py

+
+class TestTableOrientReader(object):
+
+    def test_integer(self):


yeah would be best to paramaterize these tests

jreback

small comments, otherwise lgtm.

jreback · 2018-01-04T00:19:48Z

pandas/tests/io/json/test_json_table_schema.py

        data = [1., 2., 3.]
        kinds = [pd.Series(pd.to_datetime(data), name='values'),
                 pd.to_datetime(data)]
        for kind in kinds:
-            result = make_field(kind)
+            result = convert_pandas_type_to_json_field(kind)
            expected = {"name": "values", "type": 'datetime'}
            assert result == expected

        kinds = [pd.Series(pd.to_datetime(data, utc=True), name='values'),
                 pd.to_datetime(data, utc=True)]
        for kind in kinds:


should come back later and parametrize these tests

Opened #19070 to handle this

jreback · 2018-01-04T00:20:51Z

pandas/tests/io/json/test_json_table_schema.py

+
+class TestTableOrientReader(object):
+
+    @pytest.mark.parametrize("inp,exp", [


these first 2 can just be functions, don't need to be in this class (should put next to the tests for the convert_pandas_type_to_json)

jreback · 2018-01-04T00:21:25Z

cc @TomAugspurger

TomAugspurger

This looks really good on a quick skim @WillAyd, thanks.

TomAugspurger · 2018-01-04T19:17:25Z