Skip to content

BUG-22796 Concat multicolumn tz-aware DataFrame #23036

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Oct 9, 2018
Merged

BUG-22796 Concat multicolumn tz-aware DataFrame #23036

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
9e760a0
BUG-22796 Concat multicolumn tz-aware DataFrame fails
tonytao2012 Oct 8, 2018
5ef5dd2
Fixed shape issues
tonytao2012 Oct 8, 2018
c640f69
Pass when block is not None
tonytao2012 Oct 8, 2018
5e57d77
Added whatsnew entry
tonytao2012 Oct 8, 2018
0730adf
Merge remote-tracking branch 'upstream/master' into BUG-22796
tonytao2012 Oct 8, 2018
2e89848
Fixed whitespace issues
tonytao2012 Oct 8, 2018
53c85aa
Added additional test, linked to new issue
tonytao2012 Oct 8, 2018
51ef6e5
Fixed test, explained bug in more detail for whatsnew
tonytao2012 Oct 8, 2018
54c153d
Edited whatsnew
tonytao2012 Oct 8, 2018
d6ab866
Merge branch 'master' into PR_TOOL_MERGE_PR_23036
jreback Oct 9, 2018
a0570bd
use construct_array_type
jreback Oct 9, 2018
644b349
add strict
jreback Oct 9, 2018
ce31274
handle alignment case
jreback Oct 9, 2018
93e97dc
Merge branch 'master' into PR_TOOL_MERGE_PR_23036
jreback Oct 9, 2018
de4c427
lint
jreback Oct 9, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v0.24.0.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -883,6 +883,7 @@ Reshaping
- Bug in :func:`pandas.wide_to_long` when a string is passed to the stubnames argument and a column name is a substring of that stubname (:issue:`22468`)
- Bug in :func:`merge` when merging ``datetime64[ns, tz]`` data that contained a DST transition (:issue:`18885`)
- Bug in :func:`merge_asof` when merging on float values within defined tolerance (:issue:`22981`)
- Bug in :func:`pandas.concat` when merging multicolumn DataFrames with tz-aware data (:issue`22796`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you be more specific here? The isn't an issue with all multi-column DataFrames with tz-aware data. What are the exact conditions needed?


Build Changes
^^^^^^^^^^^^^
Expand Down
7 changes: 7 additions & 0 deletions pandas/core/internals/concat.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@
from pandas.core.dtypes.cast import maybe_promote
import pandas.core.dtypes.concat as _concat

from pandas.core.indexes.datetimes import DatetimeIndex

import pandas.core.algorithms as algos


Expand Down Expand Up @@ -186,6 +188,11 @@ def get_reindexed_values(self, empty_dtype, upcasted_na):

if getattr(self.block, 'is_datetimetz', False) or \
is_datetimetz(empty_dtype):
if self.block is None:
missing_arr = np.full(self.shape, fill_value)
missing_time = DatetimeIndex(missing_arr[0],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can u walk back up the stack and see exactly where block is None- this is a guarantee iirc of this function to not be None so this might be an error higher up

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Block is none when there are a mismatched number of columns between the two concatenating dataframes. I assumed this was correct behavior?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when I last changed this, I was really trying NOT to use DTI directly here as we are trying isolate things like that to higher level methods.

See if you can revise, otherwise I will take a look.

Copy link
Contributor Author

@tonytao2012 tonytao2012 Oct 8, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, in that case, I'm not sure what to do. Clearly, block is supposed to be None in cases where the columns are mismatched:

    for blkno, placements in libinternals.get_blkno_placements(blknos,
                                                               mgr.nblocks,
                                                               group=False):

        assert placements.is_slice_like

        join_unit_indexers = indexers.copy()

        shape = list(mgr_shape)
        shape[0] = len(placements)
        shape = tuple(shape)

        if blkno == -1:
            unit = JoinUnit(None, shape)

When block is None, we have to create and return some array in get_reindexed_values, but np arrays can't have a tz dtype. I apologize if I'm missing something obvious. Feel free to take over the issue if you'd like, as I'm unsure of how to continue from here. I'll also keep thinking about it.

dtype=empty_dtype)
return missing_time
pass
elif getattr(self.block, 'is_categorical', False):
pass
Expand Down
20 changes: 20 additions & 0 deletions pandas/tests/frame/test_combine_concat.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@
import pandas.util.testing as tm
from pandas.util.testing import assert_frame_equal, assert_series_equal

import pytest


class TestDataFrameConcatCommon(TestData):

Expand Down Expand Up @@ -53,6 +55,24 @@ def test_concat_multiple_tzs(self):
expected = DataFrame(dict(time=[ts2, ts3]))
assert_frame_equal(results, expected)

@pytest.mark.parametrize('t1', ['2015-01-01',
pytest.param('pd.NaT', marks=pytest.mark.xfail(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is pd.NaT in quotes? Shouldn't it just be pd.NaT?

reason='GH23037 incorrect dtype when concatenating'))])
def test_concat_tz_NaT(self, t1):
# GH 22796
# Concating tz-aware multicolumn DataFrames
ts1 = Timestamp(t1, tz='UTC')
ts2 = Timestamp('2015-01-01', tz='UTC')
ts3 = Timestamp('2015-01-01', tz='UTC')

df1 = pd.DataFrame([[ts1, ts2]])
df2 = pd.DataFrame([[ts3]])

result = pd.concat([df1, df2])
expected = DataFrame([[ts1, ts2], [ts3, pd.NaT]], index=[0, 0])

assert_frame_equal(result, expected)

def test_concat_tuple_keys(self):
# GH 14438
df1 = pd.DataFrame(np.ones((2, 2)), columns=list('AB'))
Expand Down