Skip to content

Fix bug in contains when looking up a string in a non-monotonic datet… #13574

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 45 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
f3b114f
Fix bug in contains when looking up a string in a non-monotonic datet…
tjader Jul 6, 2016
cc0a188
BUG: Groupby.nth includes group key inconsistently #12839
adneu Jul 6, 2016
2655dae
In gbq, use googleapiclient instead of apiclient #13454 (#13458)
parthea Jul 7, 2016
f11b9c1
RLS: switch master from 0.18.2 to 0.19.0 (#13586)
jorisvandenbossche Jul 8, 2016
ba82b51
BUG: Datetime64Formatter not respecting ``formatter``
haleemur Jul 8, 2016
f95576b
BUG: Fix TimeDelta to Timedelta (#13600)
yui-knk Jul 9, 2016
5701c69
COMPAT: 32-bit compat fixes mainly in testing
jreback Jul 7, 2016
3c202b1
Added more exhaustive tests for __contains__.
tjader Jul 9, 2016
713eaa6
BUG: DatetimeIndex - Period shows ununderstandable error
sinhrks Jul 10, 2016
675a6e3
ENH: add downcast to pd.to_numeric
gfyoung Jul 10, 2016
1edc1df
CLN: remove radd workaround in ops.py
sinhrks Jul 10, 2016
2a96ab7
DEPR: rename Timestamp.offset to .freq
sinhrks Jul 10, 2016
c989570
CLN: Remove the engine parameter in CSVFormatter and to_csv
gfyoung Jun 10, 2016
c2cc68d
BUG: Block/DTI doesnt handle tzlocal properly
sinhrks Jul 10, 2016
2e8c993
BUG: Series contains NaT with object dtype comparison incorrect (#13592)
sinhrks Jul 11, 2016
5605f99
CLN/TST: Add tests for nan/nat mixed input (#13477)
sinhrks Jul 11, 2016
2f7fdd0
BUG: groupby apply on selected columns yielding scalar (GH13568) (#13…
jorisvandenbossche Jul 11, 2016
65849d3
TST: Clean up tests of DataFrame.sort_{index,values} (#13496)
IamJeffG Jul 11, 2016
8dbc0f4
DOC: asfreq clarify original NaNs are not filled (GH9963) (#13617)
jorisvandenbossche Jul 12, 2016
93b7d13
BUG: Invalid Timedelta op may raise ValueError
sinhrks Jul 12, 2016
dbd5330
CLN: Cleanup ops.py
sinhrks Jul 12, 2016
7c357d2
CLN: Removed outtype in DataFrame.to_dict (#13627)
gfyoung Jul 12, 2016
27d2915
CLN: Fix compile time warnings
yui-knk Jul 13, 2016
06103dd
Pin IPython for doc build to 4.x (see #13639)
jorisvandenbossche Jul 13, 2016
7dd4091
CLN: reorg type inference & introspection
jreback Jul 13, 2016
20de266
BLD: included pandas.api.* in setup.py (#13640)
gfyoung Jul 13, 2016
44f3229
DOC/BLD: pin IPython version to 4.2.0 (#13639) (#13647)
jorisvandenbossche Jul 14, 2016
6f0a020
TST: reorganize tools.tests (#13619)
sinhrks Jul 14, 2016
a711b42
BF(TST): allow AttributeError being raised (in addition to TypeError)…
yarikoptic Jul 14, 2016
084ceae
API, DEPR: Raise and Deprecate Reshape for Pandas Objects
gfyoung Jul 14, 2016
3f6d4bd
CLN: Fix compile time warnings
yui-knk Jul 14, 2016
c9a27ed
CLN: fix some issues in asv benchmark suite (#13630)
jorisvandenbossche Jul 14, 2016
05b976c
TST: add tests for Timestamp.toordinal/fromordinal
sinhrks Jul 15, 2016
71a0675
CLN: Initialization coincides with mapping, hence with uniqueness check
toobaz Jul 15, 2016
0a70b5f
API: Change Period('NAT') to return NaT
sinhrks Jul 15, 2016
1bee56e
BUG: construction of Series with integers on windows not default to i…
jreback Jul 15, 2016
d7c028d
CLN: Removed levels attribute from Categorical
gfyoung Jul 15, 2016
401b0ed
Fix bug in contains when looking up a string in a non-monotonic datet…
tjader Jul 6, 2016
1a86b3a
Added more exhaustive tests for __contains__.
tjader Jul 9, 2016
3bf7cce
Fix bug in contains when looking up a string in a non-monotonic datet…
tjader Jul 6, 2016
0f5a4e0
Added more exhaustive tests for __contains__.
tjader Jul 9, 2016
783ea6d
Fix bug in contains when looking up a string in a non-monotonic datet…
tjader Jul 6, 2016
592a09d
Added more exhaustive tests for __contains__.
tjader Jul 9, 2016
690e034
Fix bug in contains when looking up a string in a non-monotonic datet…
tjader Jul 6, 2016
d4348d3
Merge remote-tracking branch 'origin/bugfixes' into bugfixes
tjader Jul 16, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions pandas/tests/indexes/test_datetimelike.py
Original file line number Diff line number Diff line change
Expand Up @@ -721,6 +721,17 @@ def test_fillna_datetime64(self):
dtype=object)
self.assert_index_equal(idx.fillna('x'), exp)

def test_contains(self):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can u also add tests for TimedeltaIndex and PeriodIndex? (It looks work)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you need to just move this to the Datetimelike class which tests all datetimelike indexes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

further let's add testing with Timestamp and datetime as well (and a case where things are not there). finally add in a NaT both in the contained index as well as a tests with in

#GH13572
dates = ['2015-01-03', '2015-01-01', '2015-01-04', '2015-01-05', '2015-01-02']
monotonic = pd.to_datetime(sorted(dates))
non_monotonic = pd.to_datetime(['2015-01-03', '2015-01-01', '2015-01-04', '2015-01-05', '2015-01-02'])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can reuse dates here in the to_datetime call.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is NaT supposed to work with period indices?
The only NaT-like object I can successfully test with is this one

>>> pd.Period('NaT', freq='D') in pd.period_range('2015-01-01', periods=5, freq='D').insert(0, pd.NaT)
True

Any of pd.NaT, None, float('nan'), np.nan produce False as result, although they work on Datetimeindex objects with NaT's in them.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thx to confirm. Pls leave other nat-likes (pd.NaT, None, float('nan'), np.nan) ATM because PeriodIndex is not fully supported them yet (I'll cover in #12759, but PR is appreciated).

Copy link
Author

@tjader tjader Jul 7, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, TimedeltaIndices have a funny behavior as well. Checking for NaT-like objects always returns true

>>> None in pd.to_timedelta(range(5), unit='d') + pd.offsets.Hour(1)
True

Looks like pandas.index.TimedeltaEngine.get_loc doesn't handle NaT at all. It's probably possible to patch it on a higher level, but instinctively I feel that is where it should be handled.
Any thoughts?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sinhrks Thanks for clarifying. I've added #13582 for you.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've taken out the timedelta issues as well, and created a new PR #13603.

for idx in [non_monotonic, monotonic]:
self.assertNotIn('2015-01-06', idx)
self.assertNotIn(pd.Timestamp('2015-01-06'), idx)
for dt in reversed(dates):
self.assertIn(dt, idx)
self.assertIn(pd.Timestamp(dt), idx)

class TestPeriodIndex(DatetimeLike, tm.TestCase):
_holder = PeriodIndex
Expand Down
2 changes: 1 addition & 1 deletion pandas/tseries/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,7 @@ def _format_with_header(self, header, **kwargs):
def __contains__(self, key):
try:
res = self.get_loc(key)
return lib.isscalar(res) or type(res) == slice or np.any(res)
return lib.isscalar(res) or type(res) == slice or np.size(res)
except (KeyError, TypeError, ValueError):
return False

Expand Down