BUG: Avoid AmbiguousTimeError on groupby #14683

j-santander · 2016-11-17T22:48:08Z

TomAugspurger · 2016-11-18T02:13:13Z

Could you add your example from the issue as a test? Also a release note in the whatsnew file.

j-santander · 2016-11-18T08:45:40Z

it seems I've broken something.... I'll try to fix.

jreback · 2016-11-18T11:43:09Z

pandas/tseries/resample.py

@@ -5,6 +5,7 @@

 import pandas as pd
 from pandas.core.base import AbstractMethodError, GroupByMixin
+from pandas.core.config_init import pc_ambiguous_as_wide_doc


why are you importing this?

IDE mischieve, I'll remove.

jreback · 2016-11-18T11:44:03Z

pandas/tseries/tests/test_resample.py

-    def test_resmaple_dst_anchor(self):
+    def test_resample_across_dst(self):
+        #14682
+        df1 = DataFrame([1477786980, 1477790580], columns=['ts'])


give a 1-liner about what you are testing here

One question... where's the whatsnew file I need to add the description to?

this is a bug, so you can put in 0.19.2

jreback · 2016-11-18T11:45:54Z

pandas/tseries/resample.py

@@ -1284,8 +1285,13 @@ def _adjust_dates_anchored(first, last, offset, closed='right', base=0):
    # See https://github.com/pandas-dev/pandas/issues/8683

    first_tzinfo = first.tzinfo
+    last_tzinfo = last.tzinfo


put the issue number as a comment and a brief expl. future readers will appreciate!

jreback · 2016-11-18T11:46:38Z

pandas/tseries/index.py

@@ -439,7 +439,7 @@ def _generate(cls, start, end, periods, name, offset,
                tz = tz.localize(date.replace(tzinfo=None)).tzinfo

        if tz is not None and inferred_tz is not None:
-            if not inferred_tz == tz:
+            if not tslib.get_timezone(inferred_tz) == tslib.get_timezone(tz):


hmm, this should not be necessary; str comparison already does this

I get an exception if I don't... apparently the timezone with and without dst are not the same thing:

Traceback (most recent call last): File "C:\Users\jsantand\Code\video\be\pandas\pandas\tseries\tests\test_resample.py", line 1923, in test_resample_across_dst df.resample(rule='H').sum(), File "C:\Users\jsantand\Code\video\be\pandas\pandas\tseries\resample.py", line 541, in f return self._downsample(_method) File "C:\Users\jsantand\Code\video\be\pandas\pandas\tseries\resample.py", line 672, in _downsample self._set_binner() File "C:\Users\jsantand\Code\video\be\pandas\pandas\tseries\resample.py", line 238, in _set_binner self.binner, self.grouper = self._get_binner() File "C:\Users\jsantand\Code\video\be\pandas\pandas\tseries\resample.py", line 246, in _get_binner binner, bins, binlabels = self._get_binner_for_time() File "C:\Users\jsantand\Code\video\be\pandas\pandas\tseries\resample.py", line 661, in _get_binner_for_time return self.groupby._get_time_bins(self.ax) File "C:\Users\jsantand\Code\video\be\pandas\pandas\tseries\resample.py", line 1130, in _get_time_bins name=ax.name) File "C:\Users\jsantand\Code\video\be\pandas\pandas\util\decorators.py", line 91, in wrapper return func(*args, **kwargs) File "C:\Users\jsantand\Code\video\be\pandas\pandas\tseries\index.py", line 301, in __new__ ambiguous=ambiguous) File "C:\Users\jsantand\Code\video\be\pandas\pandas\tseries\index.py", line 444, in _generate raise AssertionError("Inferred time zone not equal to passed " AssertionError: Inferred time zone not equal to passed time zone

hmm, can you repro that directly via a DatetimeIndex construction call?

does it break anything else?

ok, I've traced the issue to the tz of input index:

I do:

df1 = DataFrame([1477786980, 1477790580], columns=['ts']) dti1 = DatetimeIndex(pd.to_datetime(df1.ts, unit='s').dt.tz_localize('UTC').dt.tz_convert('Europe/Madrid'))

Now, this dti1 index has a funny tz attribute: <DstTzInfo 'Europe/Paris' LMT+0:09:00 STD>
When resampling, the index tz is used in creating the new index and it is where the assert comes.

That probably means I don't need this change... but the fix is needed elsewhere.

Sorry, my bad.... To reproduce:

t1=Timestamp(2016,10,30,2,23).tz_localize('Europe/Madrid',ambiguous=True) t2=Timestamp(2016,10,30,2,13).tz_localize('Europe/Madrid',ambiguous=False) dti=DatetimeIndex(start=t1,end=t2,freq='H',tz='Europe/Madrid')

I've also found that the same check is done in pandas.tseries.tools.py:

def _infer_tzinfo(start, end): def _infer(a, b): tz = a.tzinfo if b and b.tzinfo: if not (tslib.get_timezone(tz) == tslib.get_timezone(b.tzinfo)): raise AssertionError('Inputs must both have the same timezone,' ' {0} != {1}'.format(tz, b.tzinfo)) return tz tz = None if start is not None: tz = _infer(start, end) elif end is not None: tz = _infer(end, start) return tz

I'll submit now the changes and see if this time they are ok.

codecov-io · 2016-11-18T20:13:06Z

Current coverage is 85.20% (diff: 100%)

Merging #14683 into master will decrease coverage by 0.08%

@@             master     #14683   diff @@
==========================================
  Files           140        143     +3   
  Lines         50706      50790    +84   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43248      43275    +27   
- Misses         7458       7515    +57   
  Partials          0          0

Powered by Codecov. Last update 45543ec...d90afaf

jreback · 2016-11-21T11:53:04Z

doc/source/whatsnew/v0.19.2.txt

@@ -43,3 +43,4 @@ Bug Fixes


 - Explicit check in ``to_stata`` and ``StataWriter`` for out-of-range values when writing doubles (:issue:`14618`)
+- Fix AmbiguousTimeError exception when resampling a DatetimeIndex in local TZ, covering a DST change (:issue:`14682`)


Bug in Ambiguous.... (use double-backticks around exception types and DatetimeIndex)

I'll change

jreback · 2016-11-21T11:56:41Z

pandas/tseries/tests/test_resample.py

+                             .dt.tz_convert('Europe/Madrid'))
+        df = DataFrame([5, 5], index=dti1)
+        assert_frame_equal(
+            df.resample(rule='H').sum(),


do

result = df.resample(....) expected = .... assert_frame_equal(result, expected)

jreback · 2016-11-21T11:57:33Z

lgtm.

cc @rockg

rockg · 2016-11-21T14:53:28Z

Looks fine.

jreback · 2016-11-22T11:30:09Z

thanks @j-santander

keep em' coming!

closes pandas-dev#14682 Author: Julian Santander <[email protected]> Author: Julian Santander <[email protected]> Closes pandas-dev#14683 from j-santander/master and squashes the following commits: d90afaf [Julian Santander] Addressing additional code inspection comments 817ed97 [Julian Santander] Addressing code inspections comments 99a5367 [Julian Santander] Fix unittest error and lint warning 940fb22 [Julian Santander] Avoid AmbiguousTimeError on groupby (cherry picked from commit 9f2e453)

Avoid AmbiguousTimeError on groupby

940fb22

Fix unittest error and lint warning

99a5367

jreback added Groupby Timezones Timezone data dtype labels Nov 18, 2016

jreback reviewed Nov 18, 2016

View reviewed changes

Addressing code inspections comments

817ed97

jreback reviewed Nov 21, 2016

View reviewed changes

jreback added the Bug label Nov 21, 2016

jreback added this to the 0.19.2 milestone Nov 21, 2016

jreback changed the title ~~Avoid AmbiguousTimeError on groupby~~ BUG: Avoid AmbiguousTimeError on groupby Nov 21, 2016

Addressing additional code inspection comments

d90afaf

jreback closed this in 9f2e453 Nov 22, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Avoid AmbiguousTimeError on groupby #14683

BUG: Avoid AmbiguousTimeError on groupby #14683

j-santander commented Nov 17, 2016 •

edited by jreback

Loading

TomAugspurger commented Nov 18, 2016

j-santander commented Nov 18, 2016

jreback Nov 18, 2016

j-santander Nov 18, 2016

jreback Nov 18, 2016

j-santander Nov 18, 2016

j-santander Nov 18, 2016

jreback Nov 18, 2016

jreback Nov 18, 2016

j-santander Nov 18, 2016

jreback Nov 18, 2016

j-santander Nov 18, 2016 •

edited

Loading

jreback Nov 18, 2016

j-santander Nov 18, 2016

j-santander Nov 18, 2016 •

edited

Loading

codecov-io commented Nov 18, 2016 •

edited

Loading

jreback Nov 21, 2016

j-santander Nov 21, 2016

jreback Nov 21, 2016

j-santander Nov 21, 2016

jreback commented Nov 21, 2016

rockg commented Nov 21, 2016

jreback commented Nov 22, 2016

		@@ -43,3 +43,4 @@ Bug Fixes


		- Explicit check in ``to_stata`` and ``StataWriter`` for out-of-range values when writing doubles (:issue:`14618`)
		- Fix AmbiguousTimeError exception when resampling a DatetimeIndex in local TZ, covering a DST change (:issue:`14682`)

BUG: Avoid AmbiguousTimeError on groupby #14683

BUG: Avoid AmbiguousTimeError on groupby #14683

Conversation

j-santander commented Nov 17, 2016 • edited by jreback Loading

TomAugspurger commented Nov 18, 2016

j-santander commented Nov 18, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

j-santander Nov 18, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

j-santander Nov 18, 2016 • edited Loading

Choose a reason for hiding this comment

codecov-io commented Nov 18, 2016 • edited Loading

Current coverage is 85.20% (diff: 100%)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Nov 21, 2016

rockg commented Nov 21, 2016

jreback commented Nov 22, 2016

j-santander commented Nov 17, 2016 •

edited by jreback

Loading

j-santander Nov 18, 2016 •

edited

Loading

j-santander Nov 18, 2016 •

edited

Loading

codecov-io commented Nov 18, 2016 •

edited

Loading