BUG: DatetimeIndex + arraylike of DateOffsets #18849

jbrockmendel · 2017-12-19T17:22:14Z

Before:

>>> dti = pd.date_range('2017-01-01', periods=2)
>>> other = np.array([pd.offsets.MonthEnd(), pd.offsets.Day(n=2)])

>>> dti + other
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: ufunc add cannot use operands with types dtype('<M8[ns]') and dtype('O')

# Same for `dti - other`, `dti + pd.Index(other)`, `dti - pd.Index(other)`

>>> dti + pd.Series(other)
0    DatetimeIndex(['2017-01-31', '2017-01-31'], dt...
1    DatetimeIndex(['2017-01-03', '2017-01-04'], dt...
dtype: object

# yikes.

After:

>>> dti + other
pandas/core/indexes/datetimelike.py:677: PerformanceWarning: Adding/subtracting array of DateOffsets to <class 'pandas.core.indexes.datetimes.DatetimeIndex'> not vectorized
  PerformanceWarning)
DatetimeIndex(['2017-01-31', '2017-01-04'], dtype='datetime64[ns]', freq=None)

>>> dti - pd.Index(other)
DatetimeIndex(['2016-12-31', '2016-12-31'], dtype='datetime64[ns]', freq=None)

>>> dti + pd.Series(other)
0   2017-01-31
1   2017-01-04
dtype: datetime64[ns]

Caveat This will need a follow-up to make sure name attribute is propogated correctly.

closes #xxxx
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

…i_add_offset_array

gfyoung · 2017-12-19T23:31:01Z

@jbrockmendel : As this PR stands, I'm a little uneasy about it. Besides the name bug that you mention as a "caveat," the fact that you have that PerformanceWarning, while not super problematic, is still a little bothersome.

gfyoung · 2017-12-19T23:31:29Z

pandas/tests/indexes/datetimes/test_arithmetic.py

+    @pytest.mark.parametrize('box', [np.array, pd.Index])
+    def test_dti_add_offset_array(self, tz, box):
+        dti = pd.date_range('2017-01-01', periods=2, tz=tz)
+        # TODO: check that `name` propogates correctly


Reference issue number below all of your new tests.

gfyoung · 2017-12-19T23:32:11Z

pandas/core/indexes/datetimelike.py

+                    return self - other[0]
+                else:
+                    from pandas.errors import PerformanceWarning
+                    warnings.warn("Adding/subtracting array of DateOffsets to "


Ah, I see that you added this yourself. Still, feel a little uneasy about this.

jbrockmendel · 2017-12-19T23:38:37Z

the fact that you have that PerformanceWarning, while not super problematic, is still a little bothersome.

This is explicitly copying the Series behavior.

gfyoung · 2017-12-19T23:43:12Z

This is explicitly copying the Series behavior.

Hmmm...I see. Theoretically, it makes sense. Though if we can avoid adding this warning, that would be great. Thus, either you can confirm that there is indeed a performance difference OR the code is rewritten so that we avoid this overhead.

jbrockmendel · 2017-12-20T00:02:20Z

Thus, either you can confirm that there is indeed a performance difference OR the code is rewritten so that we avoid this overhead

I've been working on offsets a lot recently (and upcoming, see #18854). Perf has improved quite a bit, but I can absolutely confirm that this is not a speedy operation.

jreback · 2017-12-20T11:30:52Z

pandas/core/indexes/datetimelike.py

+                if len(other) == 1:
+                    return self + other[0]
+                else:
+                    from pandas.errors import PerformanceWarning


so you are adding code here that already exists in datetimes.py:_add_offset ? is there a reason you are not dispatching to that (which may need to handle array-likes in addition to scalars). Further I am not in love with explicit type checking for DTI here. again why are you not handling that at a higher level.

_add_offset is for a single offset which does not have an implementation of apply_index. This is for an array of offsets. It's a case handled by Series but not by DatetimeIndex.

I can make a _add_offset_array and dispatch to that.

…x and PeriodIndex

…i_add_offset_array

codecov · 2017-12-23T04:46:53Z

Codecov Report

Merging #18849 into master will decrease coverage by 0.02%.
The diff coverage is 90.32%.

@@            Coverage Diff             @@
##           master   #18849      +/-   ##
==========================================
- Coverage    91.6%   91.58%   -0.03%     
==========================================
  Files         150      150              
  Lines       48939    48966      +27     
==========================================
+ Hits        44833    44845      +12     
- Misses       4106     4121      +15

Flag	Coverage Δ
#multiple	`89.94% <90.32%> (-0.03%)`	⬇️
#single	`41.72% <19.35%> (+0.55%)`	⬆️

Impacted Files	Coverage Δ
pandas/core/ops.py	`90.24% <100%> (+0.03%)`	⬆️
pandas/core/indexes/datetimes.py	`95.45% <86.66%> (-0.13%)`	⬇️
pandas/core/indexes/datetimelike.py	`97.04% <92.3%> (-0.16%)`	⬇️
pandas/core/indexes/interval.py	`92.61% <0%> (-1.21%)`	⬇️
pandas/core/sparse/array.py	`91.82% <0%> (-0.47%)`	⬇️
pandas/util/testing.py	`84.68% <0%> (-0.22%)`	⬇️
pandas/core/dtypes/cast.py	`88.42% <0%> (-0.18%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dbec3c9...81c4fbf. Read the comment docs.

jreback

lgtm. just some additional testing requested around series + offset series names.

jreback · 2017-12-23T19:53:09Z

pandas/tests/indexes/datetimes/test_arithmetic.py

+                                 name=dti.name, freq='infer')
+        tm.assert_index_equal(res, expected)
+
+    def test_dti_with_offset_series(self, tz):


can you parametrize this with name (None, same name, other name) to make sure they are propagated correctly.

Good call. Looks like Series.op(Index) was always taking on the name of the Series.

jreback · 2017-12-23T19:56:12Z

doc/source/whatsnew/v0.23.0.txt

@@ -282,6 +282,7 @@ Conversion
 - Bug in :meth:`Index.astype` with a categorical dtype where the resultant index is not converted to a :class:`CategoricalIndex` for all types of index (:issue:`18630`)
 - Bug in :meth:`Series.astype` and ``Categorical.astype()`` where an existing categorical data does not get updated (:issue:`10696`, :issue:`18593`)
 - Bug in :class:`Series` constructor with an int or float list where specifying ``dtype=str``, ``dtype='str'`` or ``dtype='U'`` failed to convert the data elements to strings (:issue:`16605`)
+- Bug in :class:`DatetimeIndex` where adding or subtracting an array-like of ``DateOffset`` objects either raised (``np.array``, ``pd.Index``) or broadcast incorrectly (``pd.Series``) (:issue:`18224`)


use this PR number here (as 18224 is a very general reference). Is there any issue for this one specifically? (don't create one, just if there is an open one).

…i_add_offset_array

…Index

…i_add_offset_array

jreback · 2017-12-29T00:25:59Z

thanks!

jbrockmendel added 5 commits December 19, 2017 08:35

implement datetimeindex ops with array of dateoffsets

01316c1

Merge branch 'master' of https://github.com/pandas-dev/pandas into dt…

81cdc41

…i_add_offset_array

add test for op against Series

179e640

remove duplicated is_offsetlike

30c7ef6

edit comments

e58849a

gfyoung added Datetime Datetime data dtype Bug labels Dec 19, 2017

gfyoung reviewed Dec 19, 2017

View reviewed changes

jreback requested changes Dec 20, 2017

View reviewed changes

jreback added the Frequency DateOffsets label Dec 20, 2017

jbrockmendel added 3 commits December 20, 2017 10:24

implement add_offset_array, sub_offset_array, tests for TimedeltaInde…

2fbebc9

…x and PeriodIndex

fixup missing import

4160c07

Merge branch 'master' of https://github.com/pandas-dev/pandas into dt…

e3c6d8e

…i_add_offset_array

jreback requested changes Dec 23, 2017

View reviewed changes

jreback reviewed Dec 23, 2017

View reviewed changes

jbrockmendel added 5 commits December 23, 2017 16:08

Merge branch 'master' of https://github.com/pandas-dev/pandas into dt…

ecb2fe9

…i_add_offset_array

Add requested test for name propagation, fi name mathcing for Series+…

0f233ba

…Index

Merge branch 'master' of https://github.com/pandas-dev/pandas into dt…

ca8c38c

…i_add_offset_array

add GH issue references to tests

d8d0af6

Merge branch 'master' of https://github.com/pandas-dev/pandas into dt…

81c4fbf

…i_add_offset_array

jbrockmendel mentioned this pull request Dec 28, 2017

BUG: fix Series[timedelta64] arithmetic with Timedelta scalars #18831

Merged

4 tasks

jreback added this to the 0.23.0 milestone Dec 29, 2017

jreback approved these changes Dec 29, 2017

View reviewed changes

jreback merged commit 7818d51 into pandas-dev:master Dec 29, 2017

jbrockmendel mentioned this pull request Dec 29, 2017

DataFrame vs Series vs Index arithmetic Roundup #18824

Closed

59 tasks

jreback mentioned this pull request Dec 29, 2017

TST: catch performance warnings #18989

Closed

hexgnu pushed a commit to hexgnu/pandas that referenced this pull request Jan 1, 2018

BUG: DatetimeIndex + arraylike of DateOffsets (pandas-dev#18849)

f8ee98b

jbrockmendel mentioned this pull request Jan 5, 2018

Fix TimedeltaIndex +/- offset array #19095

Merged

4 tasks

jbrockmendel deleted the dti_add_offset_array branch January 5, 2018 19:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: DatetimeIndex + arraylike of DateOffsets #18849

BUG: DatetimeIndex + arraylike of DateOffsets #18849

jbrockmendel commented Dec 19, 2017 •

edited

Loading

gfyoung commented Dec 19, 2017 •

edited

Loading

gfyoung Dec 19, 2017

gfyoung Dec 19, 2017

jbrockmendel commented Dec 19, 2017

gfyoung commented Dec 19, 2017

jbrockmendel commented Dec 20, 2017

jreback Dec 20, 2017

jbrockmendel Dec 20, 2017

codecov bot commented Dec 23, 2017 •

edited

Loading

jreback left a comment

jreback Dec 23, 2017

jbrockmendel Dec 24, 2017

jreback Dec 23, 2017

jreback commented Dec 29, 2017

BUG: DatetimeIndex + arraylike of DateOffsets #18849

BUG: DatetimeIndex + arraylike of DateOffsets #18849

Conversation

jbrockmendel commented Dec 19, 2017 • edited Loading

gfyoung commented Dec 19, 2017 • edited Loading

gfyoung Dec 19, 2017

Choose a reason for hiding this comment

gfyoung Dec 19, 2017

Choose a reason for hiding this comment

jbrockmendel commented Dec 19, 2017

gfyoung commented Dec 19, 2017

jbrockmendel commented Dec 20, 2017

jreback Dec 20, 2017

Choose a reason for hiding this comment

jbrockmendel Dec 20, 2017

Choose a reason for hiding this comment

codecov bot commented Dec 23, 2017 • edited Loading

Codecov Report

jreback left a comment

Choose a reason for hiding this comment

jreback Dec 23, 2017

Choose a reason for hiding this comment

jbrockmendel Dec 24, 2017

Choose a reason for hiding this comment

jreback Dec 23, 2017

Choose a reason for hiding this comment

jreback commented Dec 29, 2017

jbrockmendel commented Dec 19, 2017 •

edited

Loading

gfyoung commented Dec 19, 2017 •

edited

Loading

codecov bot commented Dec 23, 2017 •

edited

Loading