BUG: fix tzaware dataframe transpose bug #26825

jbrockmendel · 2019-06-13T04:10:08Z

This allows us to get rid of a bunch of xfails and FIXMEs in arithmetic tests. Changes in groupby are the most likely to need attention; this is not an area of the code I know well.

Not sure if TestTranspose belongs somewhere else. Suggestions welcome.

Split a couple of too-widely-scoped groupby tests.

Will follow-up with issues this closes, whatsnew entry, and GH references added to tests.

xref #23988

closes BUG: Pandas cannot create DataFrame from Numpy Array of TimeStamps #13287
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

codecov · 2019-06-13T04:58:04Z

Codecov Report

Merging #26825 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #26825      +/-   ##
==========================================
- Coverage   92.01%      92%   -0.01%     
==========================================
  Files         180      180              
  Lines       50754    50766      +12     
==========================================
+ Hits        46699    46708       +9     
- Misses       4055     4058       +3

Flag	Coverage Δ
#multiple	`90.64% <100%> (ø)`	⬆️
#single	`41.84% <35%> (-0.07%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/groupby/generic.py	`89.37% <100%> (+0.04%)`	⬆️
pandas/core/internals/construction.py	`96.04% <100%> (+0.08%)`	⬆️
pandas/io/gbq.py	`88.88% <0%> (-11.12%)`	⬇️
pandas/core/frame.py	`96.89% <0%> (-0.12%)`	⬇️
pandas/core/sorting.py	`98.35% <0%> (ø)`	⬆️
pandas/compat/_optional.py	`100% <0%> (ø)`	⬆️
pandas/core/dtypes/cast.py	`90.89% <0%> (+0.16%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a7f1d69...8b2372e. Read the comment docs.

…om2d

pandas/core/groupby/generic.py

jreback · 2019-06-13T14:33:50Z

pandas/core/internals/construction.py

@@ -160,7 +160,29 @@ def init_ndarray(values, index, columns, dtype=None, copy=False):
    # on the entire block; this is to convert if we have datetimelike's
    # embedded in an object type
    if dtype is None and is_object_dtype(values):
-        values = maybe_infer_to_datetimelike(values)
+
+        if values.ndim == 2 and values.shape[0] != 1:


this is much more messy, can we change something else to make this nicer?

Not really. I'm looking into the other places where maybe_infer_to_datetimelike is used to see if some of this can go into that. We could separate this whole block into a dedicated function. But one way or another we need to bite the bullet.

so the inside of list loop should be in pandas.core.dtypes.cast, no? (obviously up until you make the blocks themselves)

I'd like to leave this for the next pass when I'm taking a more systematic look at maybe_infer_to_datetimelike

jbrockmendel · 2019-06-13T16:39:39Z

Updated with requested edit, GH references in tests, and whatsnew.

jreback · 2019-06-13T19:24:48Z

pandas/core/groupby/generic.py

@@ -1664,3 +1655,36 @@ def _normalize_keyword_aggregation(kwargs):
        order.append((column,
                      com.get_callable_name(aggfunc) or aggfunc))
    return aggspec, columns, order
+
+
+def _recast_datetimelike_result(result):


so i would put this in pandas.core.dtypes.cast, our dumping ground for casting, right after / before maybe_infer_to-datetimelike (and you can de-privatize)

This is really kludgy code (and is replacing equally kludgy code; kind of like turtles all the way down). I'd rather keep it close to its only usage and hope it is ripped out by someone who knows this part of the code better

as i said this look a lot like maybe_convert_objects, try to use that here

Note also: we only have 5 tests that go through this path

pandas/core/groupby/generic.py

jreback · 2019-06-13T19:25:24Z

pandas/core/groupby/generic.py

+
+    Notes
+    -----
+    - Assumes Groupby._selected_obj has ndim==2 and at least one


this note doesn't seem relevant as you are passing in frame right?

That wasn't obvious to me bc were talking about the dimensions of two separate objects. Are they necessarily the same?

pandas/core/groupby/generic.py

jreback · 2019-06-13T19:26:57Z

pandas/core/internals/construction.py

@@ -160,7 +160,29 @@ def init_ndarray(values, index, columns, dtype=None, copy=False):
    # on the entire block; this is to convert if we have datetimelike's
    # embedded in an object type
    if dtype is None and is_object_dtype(values):
-        values = maybe_infer_to_datetimelike(values)
+
+        if values.ndim == 2 and values.shape[0] != 1:


so the inside of list loop should be in pandas.core.dtypes.cast, no? (obviously up until you make the blocks themselves)

…om2d

jreback · 2019-06-17T11:59:44Z

pandas/core/internals/construction.py

+
+            from pandas.core.internals.blocks import make_block
+
+            # TODO: What about re-joining object columns?


pls reuse the block creation routines below

attempts so far have broken everything. do you have a particular routine in mind?

what I mean is you can remove the create_block_manager_from_blocks and let it fall thru to 184 with I think a very small change, e.g.

if ..... blocks = bvals else: dvals = ....... blocks = [dvals]

of course pls use a longer name than dvals

jreback · 2019-06-17T12:00:07Z

pandas/core/internals/construction.py

+            # unnecessary if we ever allow 2D DatetimeArray
+
+            dvals_list = [maybe_infer_to_datetimelike(values[n, :])
+                          for n in range(len(values))]


make this a list comprehension & use enumerate if you must., this is very hard to read.

pandas/tests/arithmetic/test_datetime64.py

…om2d

jbrockmendel · 2019-06-20T15:28:12Z

@jreback anything else here actionable?

jreback · 2019-06-20T15:59:05Z

haven’t had a chance to relook

pandas/core/groupby/generic.py

jreback · 2019-06-21T01:55:46Z

pandas/core/internals/construction.py

+
+            from pandas.core.internals.blocks import make_block
+
+            # TODO: What about re-joining object columns?


what I mean is you can remove the create_block_manager_from_blocks and let it fall thru to 184 with I think a very small change, e.g.

if ..... blocks = bvals else: dvals = ....... blocks = [dvals]

of course pls use a longer name than dvals

…om2d

jbrockmendel · 2019-06-27T14:40:51Z

I think comments have been addressed.

jbrockmendel added 3 commits June 12, 2019 21:04

BUG: fix tzaware dataframe transpose bug

c9130f8

move TestTranspose

908465a

actually save

2b89d35

jbrockmendel added 3 commits June 13, 2019 07:14

Merge branch 'master' of https://github.com/pandas-dev/pandas into fr…

7bcdf16

…om2d

troubleshoot windows fails

f5759e6

Fix one more FIXME

3419983

jreback requested changes Jun 13, 2019

View reviewed changes

gfyoung added Bug DataFrame DataFrame data structure Datetime Datetime data dtype labels Jun 13, 2019

jbrockmendel added 3 commits June 13, 2019 09:34

separate out _recast_datetimelike_Result

528015e

Add GH references to tests

508f8ae

add whatsnew

c64d31f

jreback requested changes Jun 13, 2019

View reviewed changes

jbrockmendel added 3 commits June 13, 2019 13:00

annotation, typo fixup

6bd1a0a

dont alter inplace

baacaaf

Merge branch 'master' of https://github.com/pandas-dev/pandas into fr…

c23edcc

…om2d

jreback requested changes Jun 17, 2019

View reviewed changes

jbrockmendel added 5 commits June 17, 2019 10:14

Merge branch 'master' of https://github.com/pandas-dev/pandas into fr…

b559753

…om2d

use maybe_convert_objects

e39370c

xfail tests where possible

00b31e4

simplify list comprehension

0a9a886

Merge branch 'master' of https://github.com/pandas-dev/pandas into fr…

e88bc00

…om2d

jreback requested changes Jun 21, 2019

View reviewed changes

jbrockmendel added 2 commits June 24, 2019 08:44

Merge branch 'master' of https://github.com/pandas-dev/pandas into fr…

3c49874

…om2d

single assignment

5c38a76

jbrockmendel added 4 commits June 24, 2019 08:52

fall through to create_block_manager_from_blocks

657aa0c

Merge branch 'master' of https://github.com/pandas-dev/pandas into fr…

be106cc

…om2d

Fix assignment error

820c4e4

Merge branch 'master' of https://github.com/pandas-dev/pandas into fr…

8b2372e

…om2d

jreback added this to the 0.25.0 milestone Jun 27, 2019

jreback approved these changes Jun 27, 2019

View reviewed changes

jreback merged commit 27f9d05 into pandas-dev:master Jun 27, 2019

jbrockmendel deleted the from2d branch June 27, 2019 20:40

mroeschke mentioned this pull request Jul 5, 2019

BUG: maybe_infer_to_datetimelike incorrect on 2D inputs #23988

Closed

This was referenced Jan 19, 2021

BUG: 2D ndarray of dtype 'object' is always copied upon construction #39263

Closed

BUG: 2D ndarray of dtype 'object' is always copied upon construction #39272

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: fix tzaware dataframe transpose bug #26825

BUG: fix tzaware dataframe transpose bug #26825

jbrockmendel commented Jun 13, 2019 •

edited

Loading

codecov bot commented Jun 13, 2019 •

edited

Loading

jreback Jun 13, 2019

jbrockmendel Jun 13, 2019

jreback Jun 13, 2019

jbrockmendel Jun 13, 2019

jbrockmendel commented Jun 13, 2019

jreback Jun 13, 2019

jbrockmendel Jun 13, 2019

jreback Jun 13, 2019

jbrockmendel Jun 13, 2019

jreback Jun 13, 2019

jbrockmendel Jun 13, 2019

jreback Jun 13, 2019

jreback Jun 17, 2019

jbrockmendel Jun 17, 2019

jreback Jun 21, 2019

jreback Jun 17, 2019

jbrockmendel commented Jun 20, 2019

jreback commented Jun 20, 2019

jreback Jun 21, 2019

jbrockmendel commented Jun 27, 2019


		from pandas.core.internals.blocks import make_block

		# TODO: What about re-joining object columns?

BUG: fix tzaware dataframe transpose bug #26825

BUG: fix tzaware dataframe transpose bug #26825

Conversation

jbrockmendel commented Jun 13, 2019 • edited Loading

codecov bot commented Jun 13, 2019 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbrockmendel commented Jun 13, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbrockmendel commented Jun 20, 2019

jreback commented Jun 20, 2019

Choose a reason for hiding this comment

jbrockmendel commented Jun 27, 2019

jbrockmendel commented Jun 13, 2019 •

edited

Loading

codecov bot commented Jun 13, 2019 •

edited

Loading