Skip to content

Move json_normalize to pd namespace #27615

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 24 commits into from
Dec 18, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
ad922ce
Move json_normalize to pd namespace
vishwakftw Jul 26, 2019
ea57d98
Code review - I
vishwakftw Jul 28, 2019
84b0543
Fix lint
vishwakftw Jul 28, 2019
471d457
Fix failing tests due to usage of deprecated function
vishwakftw Jul 28, 2019
c28b637
Fix lint using black and isort, sorry
vishwakftw Jul 29, 2019
8b5dba4
Code Review - II
vishwakftw Jul 29, 2019
c023b68
Code Review - II
vishwakftw Jul 29, 2019
9ec93e8
Code Review - III
vishwakftw Jul 30, 2019
53ef1bf
Merge branch 'json_normalize-to-pd' of github.com:vishwakftw/pandas i…
vishwakftw Aug 5, 2019
ce1e0e3
Merge branch 'master' of https://github.com/pandas-dev/pandas into js…
vishwakftw Aug 27, 2019
94b195e
Possible doc failure fix
vishwakftw Aug 27, 2019
d8ed09e
Code Review - IV
vishwakftw Sep 13, 2019
1513bf5
Fix lint
vishwakftw Sep 13, 2019
194d1e5
Code Review - V
vishwakftw Sep 30, 2019
bc90a0d
Merge branch 'master' of https://github.com/pandas-dev/pandas into js…
vishwakftw Sep 30, 2019
c2baff4
Merge branch 'master' into json_normalize-to-pd
vishwakftw Nov 7, 2019
3af6f03
Fix lint error
vishwakftw Nov 8, 2019
7402563
Merge branch 'master' into json_normalize-to-pd
vishwakftw Nov 8, 2019
6476942
Black reformat
vishwakftw Nov 9, 2019
f415dec
Merge branch 'json_normalize-to-pd' of github.com:vishwakftw/pandas i…
vishwakftw Nov 9, 2019
746034c
Merge branch 'master' into json_normalize-to-pd
vishwakftw Nov 15, 2019
b60698f
Merge branch 'master' of https://github.com/pandas-dev/pandas into js…
vishwakftw Dec 17, 2019
7555d37
Merge branch 'json_normalize-to-pd' of github.com:vishwakftw/pandas i…
vishwakftw Dec 17, 2019
ecc64a8
Merge branch 'master' of https://github.com/pandas-dev/pandas into js…
vishwakftw Dec 18, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/redirects.csv
Original file line number Diff line number Diff line change
Expand Up @@ -777,7 +777,7 @@ generated/pandas.io.formats.style.Styler.to_excel,../reference/api/pandas.io.for
generated/pandas.io.formats.style.Styler.use,../reference/api/pandas.io.formats.style.Styler.use
generated/pandas.io.formats.style.Styler.where,../reference/api/pandas.io.formats.style.Styler.where
generated/pandas.io.json.build_table_schema,../reference/api/pandas.io.json.build_table_schema
generated/pandas.io.json.json_normalize,../reference/api/pandas.io.json.json_normalize
generated/pandas.io.json.json_normalize,../reference/api/pandas.json_normalize
generated/pandas.io.stata.StataReader.data_label,../reference/api/pandas.io.stata.StataReader.data_label
generated/pandas.io.stata.StataReader.value_labels,../reference/api/pandas.io.stata.StataReader.value_labels
generated/pandas.io.stata.StataReader.variable_labels,../reference/api/pandas.io.stata.StataReader.variable_labels
Expand Down
2 changes: 1 addition & 1 deletion doc/source/reference/io.rst
Original file line number Diff line number Diff line change
Expand Up @@ -50,13 +50,13 @@ JSON
:toctree: api/

read_json
json_normalize

.. currentmodule:: pandas.io.json

.. autosummary::
:toctree: api/

json_normalize
build_table_schema

.. currentmodule:: pandas
Expand Down
17 changes: 8 additions & 9 deletions doc/source/user_guide/io.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2136,27 +2136,26 @@ into a flat table.

.. ipython:: python

from pandas.io.json import json_normalize
data = [{'id': 1, 'name': {'first': 'Coleen', 'last': 'Volk'}},
{'name': {'given': 'Mose', 'family': 'Regner'}},
{'id': 2, 'name': 'Faye Raker'}]
json_normalize(data)
pd.json_normalize(data)

.. ipython:: python

data = [{'state': 'Florida',
'shortname': 'FL',
'info': {'governor': 'Rick Scott'},
'counties': [{'name': 'Dade', 'population': 12345},
{'name': 'Broward', 'population': 40000},
{'name': 'Palm Beach', 'population': 60000}]},
'county': [{'name': 'Dade', 'population': 12345},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason for changing this? I think OK as is was?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed it because the example became longer, which caused the lint check to fail.

{'name': 'Broward', 'population': 40000},
{'name': 'Palm Beach', 'population': 60000}]},
{'state': 'Ohio',
'shortname': 'OH',
'info': {'governor': 'John Kasich'},
'counties': [{'name': 'Summit', 'population': 1234},
{'name': 'Cuyahoga', 'population': 1337}]}]
'county': [{'name': 'Summit', 'population': 1234},
{'name': 'Cuyahoga', 'population': 1337}]}]

json_normalize(data, 'counties', ['state', 'shortname', ['info', 'governor']])
pd.json_normalize(data, 'county', ['state', 'shortname', ['info', 'governor']])

The max_level parameter provides more control over which level to end normalization.
With max_level=1 the following snippet normalizes until 1st nesting level of the provided dict.
Expand All @@ -2169,7 +2168,7 @@ With max_level=1 the following snippet normalizes until 1st nesting level of the
'Name': 'Name001'}},
'Image': {'a': 'b'}
}]
json_normalize(data, max_level=1)
pd.json_normalize(data, max_level=1)

.. _io.jsonl:

Expand Down
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v0.25.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,7 @@ which level to end normalization (:issue:`23843`):

The repr now looks like this:

.. ipython:: python
.. code-block:: ipython

from pandas.io.json import json_normalize
data = [{
Expand Down
3 changes: 3 additions & 0 deletions doc/source/whatsnew/v1.0.0.rst
100755 → 100644
Original file line number Diff line number Diff line change
Expand Up @@ -498,6 +498,9 @@ Deprecations
- The parameter ``numeric_only`` of :meth:`Categorical.min` and :meth:`Categorical.max` is deprecated and replaced with ``skipna`` (:issue:`25303`)
- The parameter ``label`` in :func:`lreshape` has been deprecated and will be removed in a future version (:issue:`29742`)
- ``pandas.core.index`` has been deprecated and will be removed in a future version, the public classes are available in the top-level namespace (:issue:`19711`)
- :func:`pandas.json_normalize` is now exposed in the top-level namespace.
Usage of ``json_normalize`` as ``pandas.io.json.json_normalize`` is now deprecated and
it is recommended to use ``json_normalize`` as :func:`pandas.json_normalize` instead (:issue:`27586`).
-

.. _whatsnew_1000.prior_deprecations:
Expand Down
2 changes: 2 additions & 0 deletions pandas/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,8 @@
read_spss,
)

from pandas.io.json import _json_normalize as json_normalize

from pandas.util._tester import test
import pandas.testing
import pandas.arrays
Expand Down
3 changes: 2 additions & 1 deletion pandas/io/json/__init__.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
from pandas.io.json._json import dumps, loads, read_json, to_json
from pandas.io.json._normalize import json_normalize
from pandas.io.json._normalize import _json_normalize, json_normalize
from pandas.io.json._table_schema import build_table_schema

__all__ = [
"dumps",
"loads",
"read_json",
"to_json",
"_json_normalize",
"json_normalize",
"build_table_schema",
]
8 changes: 7 additions & 1 deletion pandas/io/json/_normalize.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
import numpy as np

from pandas._libs.writers import convert_json_to_lines
from pandas.util._decorators import deprecate

from pandas import DataFrame

Expand Down Expand Up @@ -108,7 +109,7 @@ def nested_to_record(
return new_ds


def json_normalize(
def _json_normalize(
data: Union[Dict, List[Dict]],
record_path: Optional[Union[str, List]] = None,
meta: Optional[Union[str, List]] = None,
Expand Down Expand Up @@ -332,3 +333,8 @@ def _recursive_extract(data, path, seen_meta, level=0):
)
result[k] = np.array(v, dtype=object).repeat(lengths)
return result


json_normalize = deprecate(
"pandas.io.json.json_normalize", _json_normalize, "1.0.0", "pandas.json_normalize"
)
4 changes: 4 additions & 0 deletions pandas/tests/api/test_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,9 @@ class TestPDApi(Base):
"read_spss",
]

# top-level json funcs
funcs_json = ["json_normalize"]

# top-level to_* funcs
funcs_to = ["to_datetime", "to_numeric", "to_pickle", "to_timedelta"]

Expand Down Expand Up @@ -209,6 +212,7 @@ def test_api(self):
+ self.funcs
+ self.funcs_option
+ self.funcs_read
+ self.funcs_json
+ self.funcs_to
+ self.deprecated_funcs_in_future
+ self.deprecated_funcs
Expand Down
10 changes: 8 additions & 2 deletions pandas/tests/io/json/test_normalize.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,9 @@
import numpy as np
import pytest

from pandas import DataFrame, Index
from pandas import DataFrame, Index, json_normalize
import pandas.util.testing as tm

from pandas.io.json import json_normalize
from pandas.io.json._normalize import nested_to_record


Expand Down Expand Up @@ -698,3 +697,10 @@ def test_with_large_max_level(self):
]
output = nested_to_record(input_data, max_level=max_level)
assert output == expected

def test_deprecated_import(self):
with tm.assert_produces_warning(FutureWarning):
from pandas.io.json import json_normalize

recs = [{"a": 1, "b": 2, "c": 3}, {"a": 4, "b": 5, "c": 6}]
json_normalize(recs)