Fix NoneType error when pulling non existent field #30145

bolkedebruin · 2019-12-08T19:13:38Z

If normalizing a jsonstruct a field can be absent
due to a schema change.

closes NoneType error during json_normalize due to schema change #30148
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

WillAyd · 2019-12-09T04:44:49Z

Can you add test(s)? Typically the first thing we look for in a PR

bolkedebruin · 2019-12-09T18:25:18Z

@WillAyd of course. Test has been added.

WillAyd · 2019-12-09T18:29:10Z

Thanks for the test. I don't really agree with the change though - why wouldn't you want this to raise a TypeError?

bolkedebruin · 2019-12-09T18:32:59Z

Imagine if I have 3 records:

[{'state': 'Florida', 'info': [1, 2,]}]
[{'state': 'Texas', 'info': None}]
[{'state': 'California', 'info': [2,3,]}]

I cannot use json_normalize now with record_path=['info'] as the field for the second record is None. That's perfectly valid for this field, but not acceptable to pandas. Its a real world case btw.

if the key is not there it should raise a KeyError not a TypeError I think (I can change the test to verify for that).

WillAyd

OK I think I see the point. Few things to improve this

pandas/tests/io/json/test_normalize.py

doc/source/whatsnew/v1.0.0.rst

pandas/tests/io/json/test_normalize.py

pandas/io/json/_normalize.py

bolkedebruin · 2019-12-10T09:44:06Z

@WillAyd I have changed it to use the fixture, but I'm not entirely happy with it. The fixture sets it to float('NaN') which is different from 'None'. Now I am testing against an iterable and catching a TypeError which changes the gist of it.

What do you think?

pandas/io/json/_normalize.py

pandas/tests/io/json/test_normalize.py

bolkedebruin · 2019-12-10T22:07:48Z

@jreback @WillAyd ready for re-review

pandas/io/json/_normalize.py

jreback

looks good. ping when green.

pandas/io/json/_normalize.py

bolkedebruin · 2019-12-15T21:28:54Z

@jreback ping

pandas/io/json/_normalize.py

pandas/tests/io/json/test_normalize.py

pandas/io/json/_normalize.py

WillAyd

Glad the last thing worked; definitely makes things simpler. A few more details to hash out but going in the right direction

pandas/io/json/_normalize.py

WillAyd

lgtm. One comment that's not a blocker for me but others may disagree

Over to you @jreback

pandas/io/json/_normalize.py

pandas/tests/io/json/test_normalize.py

bolkedebruin · 2019-12-19T20:03:18Z

Ping @jreback

bolkedebruin · 2019-12-23T12:09:05Z

Friendly reminder @jreback

jreback · 2019-12-27T19:59:36Z

can you merge master and will look again

If normalizing a jsonstruct a field can be set to None due to a schema change. Co-Authored-By: William Ayd <[email protected]>

bolkedebruin · 2019-12-28T13:13:12Z

@jreback rebased. Or do you truly want me to merge master into my branch?

TomAugspurger

Merge master. There are some CI failures that should be fixed by doing that.

pandas/io/json/_normalize.py

bolkedebruin · 2019-12-31T10:19:13Z

@jreback all green again.

jreback · 2020-01-01T05:30:15Z

thanks @bolkedebruin very nice

bolkedebruin force-pushed the fix_schema_change branch 3 times, most recently from 37b4b33 to d202ead Compare December 8, 2019 22:12

bolkedebruin force-pushed the fix_schema_change branch from d202ead to 2a9d9b6 Compare December 9, 2019 18:24

WillAyd requested changes Dec 9, 2019

View reviewed changes

pandas/tests/io/json/test_normalize.py Outdated Show resolved Hide resolved

doc/source/whatsnew/v1.0.0.rst Outdated Show resolved Hide resolved

pandas/tests/io/json/test_normalize.py Outdated Show resolved Hide resolved

pandas/io/json/_normalize.py Outdated Show resolved Hide resolved

WillAyd added the IO JSON read_json, to_json, json_normalize label Dec 9, 2019

bolkedebruin force-pushed the fix_schema_change branch 2 times, most recently from 9f134b0 to 1ece957 Compare December 10, 2019 09:40

bolkedebruin requested a review from WillAyd December 10, 2019 09:44

jreback requested changes Dec 10, 2019

View reviewed changes

pandas/io/json/_normalize.py Outdated Show resolved Hide resolved

pandas/tests/io/json/test_normalize.py Show resolved Hide resolved

bolkedebruin force-pushed the fix_schema_change branch 3 times, most recently from b0cf30d to 8aef446 Compare December 10, 2019 20:18

bolkedebruin requested a review from jreback December 10, 2019 22:07

jreback requested changes Dec 11, 2019

View reviewed changes

pandas/io/json/_normalize.py Outdated Show resolved Hide resolved

pandas/io/json/_normalize.py Outdated Show resolved Hide resolved

pandas/io/json/_normalize.py Show resolved Hide resolved

bolkedebruin force-pushed the fix_schema_change branch 2 times, most recently from c93e987 to a7fd904 Compare December 13, 2019 10:34

jreback requested changes Dec 13, 2019

View reviewed changes

pandas/io/json/_normalize.py Outdated Show resolved Hide resolved

bolkedebruin force-pushed the fix_schema_change branch 5 times, most recently from f48eda9 to e17214d Compare December 15, 2019 20:50

jreback requested changes Dec 15, 2019

View reviewed changes

pandas/io/json/_normalize.py Outdated Show resolved Hide resolved

pandas/io/json/_normalize.py Show resolved Hide resolved

pandas/tests/io/json/test_normalize.py Outdated Show resolved Hide resolved

WillAyd reviewed Dec 16, 2019

View reviewed changes

pandas/io/json/_normalize.py Outdated Show resolved Hide resolved

bolkedebruin force-pushed the fix_schema_change branch 2 times, most recently from 1b489ef to 2f34c5a Compare December 16, 2019 21:04

WillAyd requested changes Dec 16, 2019

View reviewed changes

pandas/io/json/_normalize.py Show resolved Hide resolved

pandas/io/json/_normalize.py Outdated Show resolved Hide resolved

bolkedebruin force-pushed the fix_schema_change branch from 2f34c5a to e5f89e5 Compare December 16, 2019 21:53

WillAyd requested changes Dec 17, 2019

View reviewed changes

pandas/io/json/_normalize.py Outdated Show resolved Hide resolved

pandas/io/json/_normalize.py Show resolved Hide resolved

WillAyd reviewed Dec 17, 2019

View reviewed changes

pandas/io/json/_normalize.py Outdated Show resolved Hide resolved

bolkedebruin force-pushed the fix_schema_change branch 3 times, most recently from 351017f to 623fe37 Compare December 18, 2019 12:47

WillAyd approved these changes Dec 18, 2019

View reviewed changes

pandas/io/json/_normalize.py Outdated Show resolved Hide resolved

pandas/tests/io/json/test_normalize.py Outdated Show resolved Hide resolved

WillAyd added this to the 1.0 milestone Dec 18, 2019

bolkedebruin force-pushed the fix_schema_change branch from 623fe37 to 184092b Compare December 18, 2019 21:32

Fix TypeError when pulling field that is None

bcaa92c

If normalizing a jsonstruct a field can be set to None due to a schema change. Co-Authored-By: William Ayd <[email protected]>

bolkedebruin force-pushed the fix_schema_change branch from 184092b to bcaa92c Compare December 28, 2019 13:12

TomAugspurger reviewed Dec 30, 2019

View reviewed changes

pandas/io/json/_normalize.py Show resolved Hide resolved

Merge branch 'master' into fix_schema_change

4434b82

jreback approved these changes Jan 1, 2020

View reviewed changes

jreback merged commit f9fb02e into pandas-dev:master Jan 1, 2020

hweecat pushed a commit to hweecat/pandas that referenced this pull request Jan 1, 2020

Fix TypeError when pulling field that is None (pandas-dev#30145)

e749ea9

tturocy mentioned this pull request Jan 31, 2020

json_normalize in 1.0.0 with meta path specified - expects iterable #31507

Closed

charlesdong1991 mentioned this pull request Feb 11, 2020

BUG: non-iterable value in meta raise error in json_normalize #31524

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix NoneType error when pulling non existent field #30145

Fix NoneType error when pulling non existent field #30145

bolkedebruin commented Dec 8, 2019 •

edited

Loading

WillAyd commented Dec 9, 2019

bolkedebruin commented Dec 9, 2019

WillAyd commented Dec 9, 2019

bolkedebruin commented Dec 9, 2019 •

edited

Loading

WillAyd left a comment

bolkedebruin commented Dec 10, 2019

bolkedebruin commented Dec 10, 2019

jreback left a comment

bolkedebruin commented Dec 15, 2019

WillAyd left a comment

WillAyd left a comment

bolkedebruin commented Dec 19, 2019

bolkedebruin commented Dec 23, 2019

jreback commented Dec 27, 2019

bolkedebruin commented Dec 28, 2019

TomAugspurger left a comment

bolkedebruin commented Dec 31, 2019

jreback commented Jan 1, 2020

Fix NoneType error when pulling non existent field #30145

Fix NoneType error when pulling non existent field #30145

Conversation

bolkedebruin commented Dec 8, 2019 • edited Loading

WillAyd commented Dec 9, 2019

bolkedebruin commented Dec 9, 2019

WillAyd commented Dec 9, 2019

bolkedebruin commented Dec 9, 2019 • edited Loading

WillAyd left a comment

Choose a reason for hiding this comment

bolkedebruin commented Dec 10, 2019

bolkedebruin commented Dec 10, 2019

jreback left a comment

Choose a reason for hiding this comment

bolkedebruin commented Dec 15, 2019

WillAyd left a comment

Choose a reason for hiding this comment

WillAyd left a comment

Choose a reason for hiding this comment

bolkedebruin commented Dec 19, 2019

bolkedebruin commented Dec 23, 2019

jreback commented Dec 27, 2019

bolkedebruin commented Dec 28, 2019

TomAugspurger left a comment

Choose a reason for hiding this comment

bolkedebruin commented Dec 31, 2019

jreback commented Jan 1, 2020

bolkedebruin commented Dec 8, 2019 •

edited

Loading

bolkedebruin commented Dec 9, 2019 •

edited

Loading