ENH: Added to_json_schema

Lays the groundwork for pandas-dev#14386
This handles the schema part of the request there. We'll still need to
do the work to publish the data to the frontend, but that can be done
as a followup.

DOC: More notes in prose docs

Move files

use isoformat

updates

Moved to to_json

json_table

no config

refactor with classes

Added duration tests

more timedelta

Change default orient

Series test

fixup docs

JSON Table -> Table

doc

Change to table orient

added version

Handle Categorical

Many more tests

Author: TomAugspurger

doc/source/io.rst

@@ -2033,6 +2033,52 @@ using Hadoop or Spark.
   df
   df.to_json(orient='records', lines=True)
 
+
+Table Schema
+''''''''''''
+
+`Table Schema`_ is a spec for describing tabular datasets as a JSON
+object. The JSON includes information on the field names, types, and
+other attributes. You can use the orient ``'table'`` to build
+a JSON string with two fields, ``scehma`` and ``data``.
+
+.. ipython:: python
+
+   df = pd.DataFrame(
+       {'A': [1, 2, 3],
+        'B': ['a', 'b', 'c'],
+        'C': pd.date_range('2016-01-01', freq='d', periods=3),
+       }, index=pd.Index(range(3), name='idx'))
+   df
+   df.to_json(orient='table')
+
+The ``schema`` field contains the ``'fields'`` keys, which itself contains
+a list of column name to type pairs, including the Index or MultiIndex
+(see below for a list of types).
+The ``schema`` field also contains a ``primary_key`` field if the (Multi)index
+is unique.
+
+The second field, ``data``, contains the serialized data with the ``records``
+orient.
+The index is included, and any datetimes are ISO 8601 formatted, as required
+by the Table Schema spec.
+
+The full list of types supported are described in the Table Schema
+spec. This table shows the mapping from pandas types:
+
+==============  =================
+Pandas type     Table Schema type
+==============  =================
+int64           integer
+float64         number
+bool            boolean
+datetime64[ns]  date
+timedelta64[ns] duration
+object          str
+=============== =================
+
+_Table Schema: http://specs.frictionlessdata.io/json-table-schema/
+
 HTML
 ----
 

doc/source/whatsnew/v0.20.0.txt

@@ -114,6 +114,26 @@ Notably, a new numerical index, ``UInt64Index``, has been created (:issue:`14937
 - Bug in ``pd.unique()`` in which unsigned 64-bit integers were causing overflow (:issue:`14915`)
 - Bug in ``pd.value_counts()`` in which unsigned 64-bit integers were being erroneously truncated in the output (:issue:`14934`)
 
+
+Table Schema Output
+^^^^^^^^^^^^^^^^^^^
+
+The new orient ``'table'`` for :meth:`DataFrame.to_json`
+will generate a `Table Schema`_ compatible string representation of
+the data.
+
+.. ipython:: python
+
+   df = pd.DataFrame(
+       {'A': [1, 2, 3],
+        'B': ['a', 'b', 'c'],
+        'C': pd.date_range('2016-01-01', freq='d', periods=3),
+       }, index=pd.Index(range(3), name='idx'))
+   df
+   df.to_json(orient='table')
+
+.. _Table Schema: http://specs.frictionlessdata.io/json-table-schema/
+
 .. _whatsnew_0200.enhancements.other:
 
 Other enhancements

pandas/core/generic.py

@@ -1075,8 +1075,9 @@ def __setstate__(self, state):
     """
 
     def to_json(self, path_or_buf=None, orient=None, date_format='epoch',
-                double_precision=10, force_ascii=True, date_unit='ms',
-                default_handler=None, lines=False):
+                timedelta_format='epoch', double_precision=10,
+                force_ascii=True, date_unit='ms', default_handler=None,
+                lines=False):
         """
         Convert the object to a JSON string.
 
@@ -1109,10 +1110,19 @@ def to_json(self, path_or_buf=None, orient=None, date_format='epoch',
               - index : dict like {index -> {column -> value}}
               - columns : dict like {column -> {index -> value}}
               - values : just the values array
+            - ``'table'`` dict like
+              ``{'schema': [schema], 'data': [data]}``
+              where the schema component is a Table Schema
+              describing the data, and the data component is
+              like ``orient='records'``.
+
+                .. versionchanged:: 0.20.0
 
         date_format : {'epoch', 'iso'}
             Type of date conversion. `epoch` = epoch milliseconds,
-            `iso`` = ISO8601, default is epoch.
+            `iso` = ISO8601. Default is epoch, except when orient is
+            table_schema, in which case this parameter is ignored
+            and iso formatting is always used.
         double_precision : The number of decimal places to use when encoding
             floating point values, default 10.
         force_ascii : force encoded string to be ASCII, default True.
@@ -1136,6 +1146,41 @@ def to_json(self, path_or_buf=None, orient=None, date_format='epoch',
         -------
         same type as input object with filtered info axis
 
+        See Also
+        --------
+        pd.read_json
+
+        Examples
+        --------
+
+        >>> df = pd.DataFrame([['a', 'b'], ['c', 'd']],
+        ...                   index=['row 1', 'row 2'],
+        ...                   columns=['col 1', 'col 2'])
+        >>> df.to_json(orient='split')
+        '{"columns":["col 1","col 2"],
+          "index":["row 1","row 2"],
+          "data":[["a","b"],["c","d"]]}'
+
+        Encoding/decoding a Dataframe using ``'index'`` formatted JSON:
+
+        >>> df.to_json(orient='index')
+        '{"row 1":{"col 1":"a","col 2":"b"},"row 2":{"col 1":"c","col 2":"d"}}'
+
+        Encoding/decoding a Dataframe using ``'records'`` formatted JSON.
+        Note that index labels are not preserved with this encoding.
+
+        >>> df.to_json(orient='records')
+        '[{"col 1":"a","col 2":"b"},{"col 1":"c","col 2":"d"}]'
+
+        Encoding with Table Schema
+
+        >>> df.to_json(orient='table_schema')
+        '{"schema": {"fields": [{"name": "index", "type": "string"},
+                                {"name": "col 1", "type": "string"},
+                                {"name": "col 2", "type": "string"}],
+                     "primary_key": "index"},
+          "data": [{"index":"row 1","col 1":"a","col 2":"b"},
+                   {"index":"row 2","col 1":"c","col 2":"d"}]}'
         """
 
         from pandas.io import json