BUG: fix nested meta path bug (GH 27220) #27667
Conversation
|
@yanglinlee since we are dropping Python 3.5 very shortly, it may be best to skip the test on Python 3.5 by adding |
|
Do I need to make more changes to this PR? Thanks for the reviews! |
|
Fixed the merge conflict. Ping on green. |
|
@TomAugspurger The checks have passed. Now it is green. |
| @@ -287,6 +287,30 @@ def test_shallow_nested(self): | |||
| expected = DataFrame(ex_data, columns=result.columns) | |||
| tm.assert_frame_equal(result, expected) | |||
|
|
|||
| @pytest.mark.skipif(not PY36, reason="drop support for 3.5 soon") | |||
There was a problem hiding this comment.
I don't think this is needed if you follow comment below
| meta=["state", "shortname", ["info", "governor"]], | ||
| errors="ignore", | ||
| ) | ||
| ex_data = { |
There was a problem hiding this comment.
Instead of using a dict can you construct this with literal values? If you use a list of lists can circumvent 3.5 ordering issues with a dict
If you can simplify expectation would help a lot as well so as not to write out the same values 21 times
There was a problem hiding this comment.
Thanks for the suggestion. It is indeed better. I simply copied other test cases. I will change accordingly.
doc/source/whatsnew/v0.25.1.rst
Outdated
| - Avoid calling ``S3File.s3`` when reading parquet, as this was removed in s3fs version 0.3.0 (:issue:`27756`) | ||
| - Better error message when a negative header is passed in :func:`pandas.read_csv` (:issue:`27779`) | ||
| - | ||
| - Fix bug in :meth:`io.json.json_normalize` when nested meta paths with a nested record path. (:issue:`27220`) |
There was a problem hiding this comment.
Just FYI, I think we're being conservative with 0.25.2 backports. Just 3.8 compat and fixes for regressions in 0.25.
There was a problem hiding this comment.
Sounds good - @yanglinlee let's go 1.0.0 instead then
There was a problem hiding this comment.
Sounds good. This is not very urgent issue. Thanks for the reviews~
| if level + 1 > len(val): | ||
| meta_val = seen_meta[key] | ||
| # Extract the value of the key from seen_meta when |
There was a problem hiding this comment.
can you put a blank line fore these comments (and below and above), basically easier to read if they are paragraph like.
There was a problem hiding this comment.
Definitely. Will update this tomorrow.
another-green
force-pushed
the
json_improve
branch
from
16cc77e to
62ee093
Compare
|
@yanglinlee looks like a merge conflict and CI failure - mind taking a look? |
|
@yanglinlee i think the CI failure for the docbuild is unrelated. can you re-push and we'll see if its fixed |
|
@yanglinlee we would take a patch like this if you want to update. closing as stale for now. |
black pandasgit diff upstream/master -u -- "*.py" | flake8 --diffThis PR attempts to solve the missing value problems when using nested meta with nested record path.
The assumption is that we want to get the value of nested meta field even it is not on the record path. For example, in the example of GH 27220, ['info', 'governor'] and ['counties', 'name'] are not on the same path. The current behavior is to raise key error. I change the behavior to return the value of the key if we can find it in the json.
Let me know how we want to move forward with this. Thanks~