Skip to content

BUG: Concat with tz-aware and timedelta raises AttributeError #12635

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion doc/source/whatsnew/v0.18.1.txt
Original file line number Diff line number Diff line change
Expand Up @@ -125,10 +125,13 @@ Bug Fixes




- Bug in ``concat`` raises ``AttributeError`` when input data contains tz-aware datetime and timedelta (:issue:`12620`)





- Bug in ``pivot_table`` when ``margins=True`` and ``dropna=True`` where nulls still contributed to margin count (:issue:`12577`)



2 changes: 1 addition & 1 deletion pandas/core/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -2713,7 +2713,7 @@ def is_nonempty(x):
# these are mandated to handle empties as well
if 'datetime' in typs or 'datetimetz' in typs or 'timedelta' in typs:
from pandas.tseries.common import _concat_compat
return _concat_compat(to_concat, axis=axis)
return _concat_compat(to_concat, axis=axis, typs=typs)

elif 'sparse' in typs:
from pandas.sparse.array import _concat_compat
Expand Down
14 changes: 14 additions & 0 deletions pandas/tools/tests/test_merge.py
Original file line number Diff line number Diff line change
Expand Up @@ -1161,6 +1161,20 @@ def test_concat_tz_series(self):
result = pd.concat([first, second])
self.assertEqual(result[0].dtype, 'datetime64[ns, Europe/London]')

def test_concat_tz_series_with_datetimelike(self):
# GH 12620
# tz and timedelta
x = [pd.Timestamp('2011-01-01', tz='US/Eastern'),
pd.Timestamp('2011-02-01', tz='US/Eastern')]
y = [pd.Timedelta('1 day'), pd.Timedelta('2 day')]
result = concat([pd.Series(x), pd.Series(y)], ignore_index=True)
tm.assert_series_equal(result, pd.Series(x + y, dtype='object'))

# tz and period
y = [pd.Period('2011-03', freq='M'), pd.Period('2011-04', freq='M')]
result = concat([pd.Series(x), pd.Series(y)], ignore_index=True)
tm.assert_series_equal(result, pd.Series(x + y, dtype='object'))

def test_concat_period_series(self):
x = Series(pd.PeriodIndex(['2015-11-01', '2015-12-01'], freq='D'))
y = Series(pd.PeriodIndex(['2015-10-01', '2016-01-01'], freq='D'))
Expand Down
37 changes: 16 additions & 21 deletions pandas/tseries/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -238,7 +238,7 @@ class CombinedDatetimelikeProperties(DatetimeProperties, TimedeltaProperties):
__doc__ = DatetimeProperties.__doc__


def _concat_compat(to_concat, axis=0):
def _concat_compat(to_concat, axis=0, typs=None):
"""
provide concatenation of an datetimelike array of arrays each of which is a
single M8[ns], datetimet64[ns, tz] or m8[ns] dtype
Expand Down Expand Up @@ -272,38 +272,33 @@ def convert_to_pydatetime(x, axis):

return x

typs = get_dtype_kinds(to_concat)
if typs is None:
typs = get_dtype_kinds(to_concat)

# datetimetz
if 'datetimetz' in typs:

# if to_concat have 'datetime' or 'object'
# then we need to coerce to object
if 'datetime' in typs or 'object' in typs:
to_concat = [convert_to_pydatetime(x, axis) for x in to_concat]
return np.concatenate(to_concat, axis=axis)
# must be single dtype
if len(typs) == 1:

# we require ALL of the same tz for datetimetz
tzs = set([getattr(x, 'tz', None) for x in to_concat]) - set([None])
if len(tzs) == 1:
return DatetimeIndex(np.concatenate([x.tz_localize(None).asi8
for x in to_concat]),
tz=list(tzs)[0])
if 'datetimetz' in typs:
# datetime with no tz should be stored as "datetime" in typs,
# thus no need to care

# single dtype
if len(typs) == 1:
# we require ALL of the same tz for datetimetz
tzs = set([x.tz for x in to_concat])
if len(tzs) == 1:
return DatetimeIndex(np.concatenate([x.tz_localize(None).asi8
for x in to_concat]),
tz=list(tzs)[0])

if not len(typs - set(['datetime'])):
elif 'datetime' in typs:
new_values = np.concatenate([x.view(np.int64) for x in to_concat],
axis=axis)
return new_values.view(_NS_DTYPE)

elif not len(typs - set(['timedelta'])):
elif 'timedelta' in typs:
new_values = np.concatenate([x.view(np.int64) for x in to_concat],
axis=axis)
return new_values.view(_TD_DTYPE)

# need to coerce to object
to_concat = [convert_to_pydatetime(x, axis) for x in to_concat]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking at this now, I suspect this is a bug here, which I believe you have in another issue. Eg. a timedelta and a datetime (with or w/o tz) may be conveted to object, BUT they go thru the convert_to_datetime which is incorrect for timedelta.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

convert_to_pydatetime is defined in directly above, and it has a logic for timedelta. Doesn't current test case test_concat_tz_series_with_datetimelike cover your case?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this hits here: #12620


return np.concatenate(to_concat, axis=axis)