NoneType error during json_normalize due to schema change #30148
Comments
5 tasks
|
Can you post of a small example json_struct that reproduces the problem |
|
Imagine if I have 3 records: I cannot use json_normalize now with record_path=['info'] as the field for the second record is None. That's perfectly valid for this field, but not acceptable to pandas. Its a real world case btw. if the key is not there it should raise a KeyError not a TypeError I think (I can change the test to verify for that). |
|
@bolkedebruin thank you for providing an example, but i need you to go a little bit further and make it into something i can copy/paste to reproduce the problem (https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports) cc @WillAyd |
|
@jbrockmendel please see the PR. I've written a test for it. |
|
excellent, thank you |
LTe
added a commit
to LTe/pandas
that referenced
this issue
Apr 16, 2020
When `record_path` will points to something that is Iterable but is not
a sequence in JSON world we will receive odd results.
```
>>> json_normalize([{'key': 'value'}], record_path='key')
0
0 v
1 a
2 l
3 u
4 e
```
Based on RFC 8259 (https://tools.ietf.org/html/rfc8259) a JSON value MUST be an
object, array, number, or string, false, null, true. But only two of
they should be treated as Iterable.
```
An object is an unordered *collection* of zero or more name/value
pairs, where a name is a string and a value is a string, number,
boolean, null, object, or array.
An array is an ordered *sequence* of zero or more values.
--
https://tools.ietf.org/html/rfc8259#page-3
```
Based on that `[{'key':'value'}]` and `{'key':'value'}` should not be
treated in the same way. In `json_normalize` documentation `record_path`
is described as `Path in each object to list of records`.
So when we want to translate JSON to python like an object we need to take
into consideration list (sequence). Based on that `record_path` should
point out to `list`, not `Iterable`.
In specs I added all possibile values that are allowed in JSON and
should not be treated as collection. There is a special case for null
value that is already implemented.
+--------+---------+----------+---------------------------+
| type | value | Iterable | Should be treated as list |
+--------+---------+----------+---------------------------+
| object | {} | Yes | No (unordered list) |
| array | [] | Yes | Yes |
| number | 1 | No | No |
| string | "value" | Yes | No |
| false | False | No | No |
| null | Null | No | No (Check pandas-dev#30148) |
| true | True | No | No |
+--------+---------+----------+---------------------------+
LTe
added a commit
to LTe/pandas
that referenced
this issue
Apr 16, 2020
When `record_path` will points to something that is Iterable but is not
a sequence in JSON world we will receive odd results.
```
>>> json_normalize([{'key': 'value'}], record_path='key')
0
0 v
1 a
2 l
3 u
4 e
```
Based on RFC 8259 (https://tools.ietf.org/html/rfc8259) a JSON value MUST be an
object, array, number, or string, false, null, true. But only two of
they should be treated as Iterable.
```
An object is an unordered *collection* of zero or more name/value
pairs, where a name is a string and a value is a string, number,
boolean, null, object, or array.
An array is an ordered *sequence* of zero or more values.
--
https://tools.ietf.org/html/rfc8259#page-3
```
Based on that `[{'key':'value'}]` and `{'key':'value'}` should not be
treated in the same way. In `json_normalize` documentation `record_path`
is described as `Path in each object to list of records`.
So when we want to translate JSON to python like an object we need to take
into consideration list (sequence). Based on that `record_path` should
point out to `list`, not `Iterable`.
In specs I added all possibile values that are allowed in JSON and
should not be treated as collection. There is a special case for null
value that is already implemented.
| type | value | Iterable | Should be treated as list |
|--------|---------|----------|---------------------------|
| object | {} | Yes | No (unordered list) |
| array | [] | Yes | Yes |
| number | 1 | No | No |
| string | "value" | Yes | No |
| false | False | No | No |
| null | Null | No | No (Check pandas-dev#30148) |
| true | True | No | No |
5 tasks
LTe
added a commit
to LTe/pandas
that referenced
this issue
Apr 16, 2020
When `record_path` points to something that is Iterable but is not
a sequence in JSON world we will receive odd results.
```
>>> json_normalize([{'key': 'value'}], record_path='key')
0
0 v
1 a
2 l
3 u
4 e
```
Based on RFC 8259 (https://tools.ietf.org/html/rfc8259) a JSON value MUST be
object, array, number, or string, false, null, true. But only two of them
should be treated as Iterable.
```
An object is an unordered *collection* of zero or more name/value
pairs, where a name is a string and a value is a string, number,
boolean, null, object, or array.
An array is an ordered *sequence* of zero or more values.
--
https://tools.ietf.org/html/rfc8259#page-3
```
Based on that `[{'key': 'value'}]` and `{'key': 'value'}` should not be
treated in the same way. In `json_normalize` documentation `record_path`
is described as `Path in each object to list of records`.
So when we want to translate JSON to Python like an object we need to take
into consideration a list (sequence). Based on that `record_path` should
point out to `list`, not `Iterable`.
In specs I added all possibile values that are allowed in JSON and
should not be treated as a collection. There is a special case for null
value that is already implemented.
| type | value | Iterable | Should be treated as list |
|--------|---------|----------|---------------------------|
| object | {} | Yes | No (unordered list) |
| array | [] | Yes | Yes |
| number | 1 | No | No |
| string | "value" | Yes | No |
| false | False | No | No |
| null | Null | No | No (Check pandas-dev#30148) |
| true | True | No | No |
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Code sample
Problem description
Normalizing a json with an absent field at a certain point in time due to a schema change(s) results in a NoneType error
Expected Output
Continue normalization
Output of
pd.show_versions()INSTALLED VERSIONS
commit : 37b4b33
python : 3.7.5.final.0
python-bits : 64
OS : Darwin
OS-release : 19.0.0
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.UTF-8
pandas : 0.26.0.dev0+1247.g37b4b3307.dirty
numpy : 1.17.4
pytz : 2019.3
dateutil : 2.8.1
pip : 19.2.3
setuptools : 41.2.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
The text was updated successfully, but these errors were encountered: