[bug] don't remove timezone-awareness when using the method from Dat…

[MarcoGorelli]

…aFrame

• closes Appending to DataFrame removes the timezone-awareness #30238
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[jreback]

append is overly complicated and duplicating many parts of concat.

I think you might be able to remove almost all of 6751-6771, e.g. once its a Series . All of the column stuff is not necessary (I could be wrong but give this a try)

IOW this does the right thing

In [17]: df = pd.DataFrame({'A': pd.date_range('20130101', periods=3, tz='UTC'), 'B': range(3)})                                                                                                                            

In [18]: pd.concat([df, df.iloc[0].to_frame().T.infer_objects()]).dtypes                                                                                                                                                    
Out[18]: 
A    datetime64[ns, UTC]
B                  int64
dtype: object

[MarcoGorelli]

All of the column stuff is not necessary (I could be wrong but give this a try)

@jreback Tried this, but it seems it's necessary for pandas/tests/reshape/test_concat.py::TestAppend::test_append_same_columns_type

[mroeschke]

@MarcoGorelli this change solved the original issue. Maybe give this a shot:

(pandas-dev) matthewroeschke:pandas-mroeschke matthewroeschke$ git diff
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
index 766437dba..62391b7cc 100644
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -6771,7 +6771,7 @@ class DataFrame(NDFrame):
                 combined_columns = self.columns.astype(object).append(idx_diff)
             other = other.reindex(combined_columns, copy=False)
             other = DataFrame(
-                other.values.reshape((1, len(other))),
+                other._values,
                 index=index,
                 columns=combined_columns,
             )

[MarcoGorelli]

@mroeschke I would still need to reshape though, wouldn't I? Else I don't see how

pandas/tests/frame/test_combine_concat.py::TestDataFrameConcatCommon::test_append_series_dict

would pass.

But if I do reshape, then pandas/tests/frame/test_combine_concat.py::TestDataFrameConcatCommon::test_append_timestamps_aware_or_naive fails with

AttributeError: 'DatetimeArray' object has no attribute 'reshape'

And I can't just pass the dtype directly to the constructor, else I'm blocked by #30428

[mroeschke]

Extension arrays (DatetimeArray) are still essentially 1D only right now, so reshaping is not possible. As you can see in the test_append_series_dict it's failing the verify_integrety=True test which I think are not really well integrated with extension arrays.

[jreback]

small changes. I would also like to test this with categorical, nullable integer and other extension types (likely this fixes).

so if you can file an issue for this (and followup PR would be great).

can also do here if you'd like alternately

[MarcoGorelli]

@mroeschke Thanks for your explanation. What would you like me to do then, use ._value and change test_append_series_dict, or keep as is?

[MarcoGorelli]

@jreback thanks for your review, I've made those changes.

I've opened up another issue for the other dtypes, will work on that next week if no one else has taken it up by then

[jreback]

thanks!