Skip to content

BUG: dropna() on single column timezone-aware values (#13407) #20422

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 25, 2018

Conversation

JQGoh
Copy link
Contributor

@JQGoh JQGoh commented Mar 20, 2018

As mentioned by the title of #13407, DataFrame.values not a 2D-array when constructed from timezone-aware datetimes. Hence, notna(frame.values) raises ValueErorr: 'axis' entry is out of bounds because user tries to apply dropna on a 1D-array.

I mainly include the example mentioned by the original author of #13407. Let me know if further test examples are required for this issue. Thanks.

Edit: I add additional test example to demonstrate the dropping of np.nan and None from the Series having timezone-ware datetimes.

@codecov
Copy link

codecov bot commented Mar 20, 2018

Codecov Report

Merging #20422 into master will decrease coverage by 0.02%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #20422      +/-   ##
==========================================
- Coverage   91.85%   91.82%   -0.03%     
==========================================
  Files         152      152              
  Lines       49231    49232       +1     
==========================================
- Hits        45220    45209      -11     
- Misses       4011     4023      +12
Flag Coverage Δ
#multiple 90.21% <100%> (-0.03%) ⬇️
#single 41.88% <100%> (+0.04%) ⬆️
Impacted Files Coverage Δ
pandas/core/frame.py 97.18% <100%> (ø) ⬆️
pandas/plotting/_converter.py 65.07% <0%> (-1.74%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b26f0f7...f995d93. Read the comment docs.

@pep8speaks
Copy link

pep8speaks commented Mar 23, 2018

Hello @JQGoh! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on March 25, 2018 at 14:23 Hours UTC

@JQGoh JQGoh force-pushed the GH13407 branch 3 times, most recently from 5445316 to 4fee776 Compare March 23, 2018 22:37
@JQGoh
Copy link
Contributor Author

JQGoh commented Mar 23, 2018

@TomAugspurger In case this could be of your interest. I recall that my first commit passed all the CI tests. However, the current failure is related to the new change in pandas/tests/extension/json/test_json.py

@TomAugspurger
Copy link
Contributor

Sorry about that. Fixed on master.

If you fetch master and merge it into your branch things should pass. You can also fixup that pep8 issue.

@JQGoh
Copy link
Contributor Author

JQGoh commented Mar 25, 2018

@TomAugspurger Thank you for the revisions. I have fixed the PEP8 issues as well, please help to review the changes if possible.

@jreback jreback added Bug Timezones Timezone data dtype Reshaping Concat, Merge/Join, Stack/Unstack, Explode labels Mar 25, 2018
Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. some tests comments. ping on green.

@@ -183,6 +186,27 @@ def test_dropna_multiple_axes(self):
inp.dropna(how='all', axis=(0, 1), inplace=True)
assert_frame_equal(inp, expected)

def test_dropna_tz_aware_datetime(self):
# GH13407
# Example reported by GH13407
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you remove this second coment

assert_frame_equal(result, expected)

# Ex2
df2 = DataFrame({'Time': [dt1, None, np.nan, dt2]})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

name this df, and use result and expected here

@JQGoh
Copy link
Contributor Author

JQGoh commented Mar 25, 2018

@jreback Thank you for your suggestions. Done.

@jreback jreback added this to the 0.23.0 milestone Mar 25, 2018
@jreback jreback merged commit add37ac into pandas-dev:master Mar 25, 2018
@jreback
Copy link
Contributor

jreback commented Mar 25, 2018

thanks for the patch @JQGoh nice to have an older issue closed!

javadnoorb pushed a commit to javadnoorb/pandas that referenced this pull request Mar 29, 2018
dworvos pushed a commit to dworvos/pandas that referenced this pull request Apr 2, 2018
kornilova203 pushed a commit to kornilova203/pandas that referenced this pull request Apr 23, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Reshaping Concat, Merge/Join, Stack/Unstack, Explode Timezones Timezone data dtype
Projects
None yet
Development

Successfully merging this pull request may close these issues.

DataFrame.values not a 2D-array when constructed from timezone-aware datetimes
4 participants