Skip to content

BUG: non-iterable value in meta raise error in json_normalize #31524

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 33 commits into from
Mar 11, 2020
Merged

BUG: non-iterable value in meta raise error in json_normalize #31524

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
7e461a1
remove \n from docstring
charlesdong1991 Dec 3, 2018
1314059
fix conflicts
charlesdong1991 Jan 19, 2019
8bcb313
Merge remote-tracking branch 'upstream/master'
charlesdong1991 Jul 30, 2019
24c3ede
Merge remote-tracking branch 'upstream/master'
charlesdong1991 Jan 14, 2020
dea38f2
fix issue 17038
charlesdong1991 Jan 14, 2020
cd9e7ac
revert change
charlesdong1991 Jan 14, 2020
e5e912b
revert change
charlesdong1991 Jan 14, 2020
2d21d1e
Merge remote-tracking branch 'upstream/master' into fix_issue_31507
charlesdong1991 Jan 31, 2020
fcb4b80
fix uo
charlesdong1991 Jan 31, 2020
8ec4450
pep8
charlesdong1991 Jan 31, 2020
a33d05c
whatsnew
charlesdong1991 Jan 31, 2020
6bedc52
fix up
charlesdong1991 Feb 1, 2020
1f0f3bc
fixup
charlesdong1991 Feb 1, 2020
ce81951
Merge remote-tracking branch 'upstream/master' into fix_issue_31507
charlesdong1991 Feb 2, 2020
5de348c
move around
charlesdong1991 Feb 2, 2020
3c38c48
better python
charlesdong1991 Feb 11, 2020
130d71b
fix conflict
charlesdong1991 Feb 11, 2020
3ef920f
fixup
charlesdong1991 Feb 11, 2020
0b46239
fixup
charlesdong1991 Feb 11, 2020
f25a4f4
rebase and fix conflict
charlesdong1991 Feb 16, 2020
6eee937
clearer python
charlesdong1991 Feb 23, 2020
d4d9218
Merge remote-tracking branch 'upstream/master' into fix_issue_31507
charlesdong1991 Feb 23, 2020
a23eb2d
rebase and resolve conflict
charlesdong1991 Mar 4, 2020
4c5d61b
rename methods
charlesdong1991 Mar 4, 2020
9726014
change typing
charlesdong1991 Mar 4, 2020
67a43fe
fix annotation
charlesdong1991 Mar 4, 2020
392e3d1
change back to any
charlesdong1991 Mar 4, 2020
011dbb0
change back to scalar
charlesdong1991 Mar 4, 2020
3e74a3a
Merge remote-tracking branch 'upstream/master' into fix_issue_31507
charlesdong1991 Mar 5, 2020
9476af7
fixup
charlesdong1991 Mar 10, 2020
c399983
Merge remote-tracking branch 'upstream/master' into fix_issue_31507
charlesdong1991 Mar 10, 2020
6165467
add ignore type
charlesdong1991 Mar 10, 2020
7a20b8c
fix annotation
charlesdong1991 Mar 11, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions doc/source/whatsnew/v1.0.1.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,8 @@ I/O

-
-
- Bug in :meth:`pandas.json_normalize` when value in meta path is not iterable (:issue:`31507`)


Plotting
^^^^^^^^
Expand Down
9 changes: 6 additions & 3 deletions pandas/io/json/_normalize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -230,15 +230,18 @@ def _json_normalize(
Returns normalized data with columns prefixed with the given string.
"""

def _pull_field(js: Dict[str, Any], spec: Union[List, str]) -> Iterable:
def _pull_field(
js: Dict[str, Any], spec: Union[List, str], is_meta: bool = True
) -> Any:
result = js # type: ignore
if isinstance(spec, list):
for field in spec:
result = result[field]
else:
result = result[spec]

if not isinstance(result, Iterable):
# GH 31507 iterable limit should only be used on record, not meta
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a code smell to add this. so it seems that result can be null, iterabe, or a scalar? if its a scalar what is the return value here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you mean [99]? then will still return 99

i think this Iterable should only restrict if specifying record_path, but not for meta? Am I right about the behaviour here @WillAyd ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do agree that this is getting a little strange, especially since there is inspection of meta on line 273 of the same module and we are essentially repeating that here with a boolean indicator being manually supplied

@charlesdong1991 do you see a way to more logically order this function so we don't have to use this bool indicator?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for your reply! @WillAyd

I will think about it a bit, i feel the patch for that Iterable in this _pull_field has code smell a bit because it is used for two cases which have different requirements

Copy link
Member Author

@charlesdong1991 charlesdong1991 Feb 4, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

especially since there is inspection of meta on line 273 of the same module and we are essentially repeating that here with a boolean indicator being manually supplied

i just take a look again at the current codebase, seems that line 273 and onwards you referred is to validate/transform the key which is assigned to meta not the value the key associated with, in this case, key is id (and we could either specify it as [id] or id or more complex cases with nested list, and this part of code will deal with it), however, the code in _pull_field is to pull the value out, and in this case, the value is 99, and for record_path the value should be an Iterable, while for meta it does not necessarily be the case, and therefore I think the patch added to check the type should only work for values of which the element of record_path point to, not for meta, maybe I miss some functionalities of either of them? @WillAyd @jreback

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@charlesdong1991 does splitting this into separate functions for record_path vs meta and just using this as a base for those functions make things cleaner?

if not isinstance(result, Iterable) and not is_meta:
if pd.isnull(result):
result = [] # type: ignore
else:
Expand Down Expand Up @@ -296,7 +299,7 @@ def _recursive_extract(data, path, seen_meta, level=0):
_recursive_extract(obj[path[0]], path[1:], seen_meta, level=level + 1)
else:
for obj in data:
recs = _pull_field(obj, path[0])
recs = _pull_field(obj, path[0], is_meta=False)
recs = [
nested_to_record(r, sep=sep, max_level=max_level)
if isinstance(r, dict)
Expand Down
10 changes: 10 additions & 0 deletions pandas/tests/io/json/test_normalize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -749,3 +749,13 @@ def test_series_non_zero_index(self):
}
)
tm.assert_frame_equal(result, expected)

def test_meta_non_iterable(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you move this near the other test

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved! and wait for @WillAyd comment on the other one

# GH 31507
data = """[{"id": 99, "data": [{"one": 1, "two": 2}]}]"""

result = json_normalize(json.loads(data), record_path=["data"], meta=["id"])
expected_values = [[1, 2, "99"]]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The expected value here should be 99 not "99" (probably cause of CI error)

columns = ["one", "two", "id"]
expected = DataFrame(expected_values, columns=columns)
tm.assert_frame_equal(result, expected)