Skip to content

Commit d400c63

Browse files
committed
BUG: Raise a TypeError when record_path doesn't point to an array
When `record_path` will points to something that is Iterable but is not a sequence in JSON world we will receive odd results. ``` >>> json_normalize([{'key': 'value'}], record_path='key') 0 0 v 1 a 2 l 3 u 4 e ``` Based on RFC 8259 (https://tools.ietf.org/html/rfc8259) a JSON value MUST be an object, array, number, or string, false, null, true. But only two of they should be treated as Iterable. ``` An object is an unordered *collection* of zero or more name/value pairs, where a name is a string and a value is a string, number, boolean, null, object, or array. An array is an ordered *sequence* of zero or more values. -- https://tools.ietf.org/html/rfc8259#page-3 ``` Based on that `[{'key':'value'}]` and `{'key':'value'}` should not be treated in the same way. In `json_normalize` documentation `record_path` is described as `Path in each object to list of records`. So when we want to translate JSON to python like an object we need to take into consideration list (sequence). Based on that `record_path` should point out to `list`, not `Iterable`. In specs I added all possibile values that are allowed in JSON and should not be treated as collection. There is a special case for null value that is already implemented. +--------+---------+----------+---------------------------+ | type | value | Iterable | Should be treated as list | +--------+---------+----------+---------------------------+ | object | {} | Yes | No (unordered list) | | array | [] | Yes | Yes | | number | 1 | No | No | | string | "value" | Yes | No | | false | False | No | No | | null | Null | No | No (Check pandas-dev#30148) | | true | True | No | No | +--------+---------+----------+---------------------------+
1 parent 64de8f4 commit d400c63

File tree

3 files changed

+14
-11
lines changed

3 files changed

+14
-11
lines changed

doc/source/whatsnew/v1.1.0.rst

+1
Original file line numberDiff line numberDiff line change
@@ -529,6 +529,7 @@ I/O
529529
- Bug in :meth:`DataFrame.to_sql` where an ``AttributeError`` was raised when saving an out of bounds date (:issue:`26761`)
530530
- Bug in :meth:`read_excel` did not correctly handle multiple embedded spaces in OpenDocument text cells. (:issue:`32207`)
531531
- Bug in :meth:`read_json` was raising ``TypeError`` when reading a list of booleans into a Series. (:issue:`31464`)
532+
- Bug in :func:`pandas.io.json.json_normalize` where location specified by `record_path` doesn't point to an array. (:issue:`26284`)
532533

533534
Plotting
534535
^^^^^^^^

pandas/io/json/_normalize.py

+6-6
Original file line numberDiff line numberDiff line change
@@ -239,23 +239,23 @@ def _pull_field(
239239
result = result[spec]
240240
return result
241241

242-
def _pull_records(js: Dict[str, Any], spec: Union[List, str]) -> Iterable:
242+
def _pull_records(js: Dict[str, Any], spec: Union[List, str]) -> List:
243243
"""
244244
Interal function to pull field for records, and similar to
245-
_pull_field, but require to return Iterable. And will raise error
245+
_pull_field, but require to return list. And will raise error
246246
if has non iterable value.
247247
"""
248248
result = _pull_field(js, spec)
249249

250-
# GH 31507 GH 30145, if result is not Iterable, raise TypeError if not
250+
# GH 31507 GH 30145, GH 26284 if result is not list, raise TypeError if not
251251
# null, otherwise return an empty list
252-
if not isinstance(result, Iterable):
252+
if not isinstance(result, list):
253253
if pd.isnull(result):
254254
result = []
255255
else:
256256
raise TypeError(
257-
f"{js} has non iterable value {result} for path {spec}. "
258-
"Must be iterable or null."
257+
f"{js} has non list value {result} for path {spec}. "
258+
"Must be list or null."
259259
)
260260
return result
261261

pandas/tests/io/json/test_normalize.py

+7-5
Original file line numberDiff line numberDiff line change
@@ -475,13 +475,15 @@ def test_nonetype_record_path(self, nulls_fixture):
475475
expected = DataFrame({"i": 2}, index=[0])
476476
tm.assert_equal(result, expected)
477477

478-
def test_non_interable_record_path_errors(self):
479-
# see gh-30148
480-
test_input = {"state": "Texas", "info": 1}
478+
@pytest.mark.parametrize("value", ["false", "true", "{}", "1", '"text"'])
479+
def test_non_list_record_path_errors(self, value):
480+
# see gh-30148, GH 26284
481+
parsed_value = json.loads(value)
482+
test_input = {"state": "Texas", "info": parsed_value}
481483
test_path = "info"
482484
msg = (
483-
f"{test_input} has non iterable value 1 for path {test_path}. "
484-
"Must be iterable or null."
485+
f"{test_input} has non list value {parsed_value} for path {test_path}. "
486+
"Must be list or null."
485487
)
486488
with pytest.raises(TypeError, match=msg):
487489
json_normalize([test_input], record_path=[test_path])

0 commit comments

Comments
 (0)