DEPR: DataFrame.groupby(axis=1)

[jbrockmendel]

It isn't obvious to me what you can do with df.groupby(axis=1) that you can't do with df.T.groupby(axis=0). In many cases we do the transpose under the hood anyway, so it isn't saving much/anything perf-wise.

[rhshadrach]

There are only two benefits I see with keeping the axis argument.

1. Wrapping the operation with transpose may be described as a clever trick that is likely not apparent to new users.
2. dtypes aren't coerced to object (I'd guess they are coerced to object internally and then fixed, but I haven't checked)

For 1, this can at least partially be mitigated by adding a section to the User Guide. For 2, users can call infer_objects, although they may not realize the coercion happened.

I don't think either of these justify keeping the axis argument. I'm +1 on deprecating.

[DeaMariaLeon]

I don’t understand why we would remove the axis keyword from the groupby method.
I’m personally using code that groups columns using this option.
Sure, we can transpose, group by the index and transpose again but this is not convenient for the user.
Besides, I fail to see why groupby would be different from the many other methods that have an axis option. Why for instance would we remove it from groupby while keeping it in sum?

[jbrockmendel]

Besides, I fail to see why groupby would be different from the many other methods that have an axis option. Why for instance would we remove it from groupby while keeping it in sum?

The main difference is maintenance burden. DataFrame.sum(axis=1) is pretty simple, the cause of few bugs, and easy to debug when one is found. DataFrameGroupBy with axis=1 is the opposite.

[rhshadrach]

@DeaMariaLeon - how often do you find yourself using axis=1 in groupby? Could you share an example, or some examples? In particular, it'd be good to know how the groups are determined (a list?), the number of groups, and what operations you find useful applying to them.

[DeaMariaLeon]

@rhshadrach here is an example.

It is quite a common need to group by axis=1 when analyzing coorporate financial data. I used to have this: column names as months, rows were the different type of costs. Then I needed to find total costs by quarters and/or years.

This example is a bit different:
groupby example.pdf

[rhshadrach]

Thanks for the example @DeaMariaLeon. In your case, I would also consider separating out the three sets of columns, computing via .sum(axis=1), and then combining via pd.concat([...], axis=1). I agree this is more verbose than what is possible using groupby with axis=1. However, we also need to consider the number of functions that groupby must support, for example, here is an odd one that currently gives incorrect ouptut:

df = pd.DataFrame({'a': [1, 1, 2], 'b': [3, 4, 5], 'c': [6, 7, 8]})
print(df.groupby([0, 0, 1], axis=1).describe())
#      count      mean      std  min  25%  50%  75%  max
# 0 a    3.0  1.333333  0.57735  1.0  1.0  1.0  1.5  2.0
#   b    3.0  4.000000  1.00000  3.0  3.5  4.0  4.5  5.0
# 1 c    3.0  7.000000  1.00000  6.0  6.5  7.0  7.5  8.0

In my opinion, this is not worth the effort it would take to fix. Aside from deprecating axis=1, the alternatives here are not allowing some ops to have axis=1 or just not fixing bugs like these. I find neither of these to be palatable.

[DeaMariaLeon]

Thanks for the explanations @rhshadrach and @jbrockmendel.

[rhshadrach]

For visibility and any thoughts: cc @pandas-dev/pandas-core

[MarcoGorelli]

Is there a performance hit to doing group by with axis 1?

If so, then in addition to the arguments already listed, I think it makes a case for deprecating, so that users are nudged towards more performant ways

[jbrockmendel]

Is there a performance hit to doing group by with axis 1?

In the axis=1 case most groupby methods either a) do the transpose in _get_data_to_aggregate or b) go through a non-cython path. Especially if you are doing many aggregations you're better off doing the one transpose up front.

But I haven't run any timings.

[Dr-Irv]

I did some experimentation using the example provided. Seems like using groupby(axis=1) is 3x faster:

import pandas as pd
import numpy as np

items = ["A_10", "A_20", "A_30", "B_10", "B_20", "B_30", "C_10", "C_20", "C_30"]

costs = np.random.default_rng().uniform(low=1, high=10_000, size=(1000, len(items)))

df = pd.DataFrame(costs, columns=items)

def grouper(c):
    if c.startswith("A"):
        return "ACCESS"
    if c.startswith("B"):
        return "BACKHAUL"
    if c.startswith("C"):
        return "CORE"

def transit(df):
    return (
        df.T.rename_axis(index=["colname"])
        .reset_index()
        .assign(group=lambda df: df["colname"].map(grouper))
        .drop(columns=["colname"])
        .groupby("group")
        .sum()
        .T
    )

def aggit(df):
    return df.groupby(by=grouper, axis=1).sum()

%timeit transit(df)

2.96 ms ± 380 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit aggit(df)
931 µs ± 14.1 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

That's a factor of 3 faster using groupby(axis=1). I think the multiple .T operations can be expensive. But maybe there's a faster way of doing it.

[WillAyd]

I am also +1 to deprecating axis. I feel like pandas is in a weird state where we act columnar in 99% of use cases but aren't explicitly committed to that.

I've often wondered if we wouldn't be better off removing axis everywhere and having users transpose when they want that behavior

[rhshadrach]

@Dr-Irv - I think your transit can be just return df.T.groupby(grouper).sum().T. With this, I get slight faster timings with transit (364 µs for transit vs 389 µs for aggit).

[Dr-Irv]

@Dr-Irv - I think your transit can be just return df.T.groupby(grouper).sum().T. With this, I get slight faster timings with transit (364 µs for transit vs 389 µs for aggit).

Agreed. I am seeing 832 µs vs. 931 µs . Thanks for pointing out the better use of the grouper

[ryan-williams]

groupby(axis=1) helped me today, disappointed to see that it's deprecated.

Here's a full repro of my use case, including one workaround involving 4 transpose calls and another involving 2 transposes and some logic changes.

The 4-transpose version is simplest; .T is inserted:

1. before calling .groupby
2. at the start of the function passed to DataFrameGroupBy.apply
3. at the end of the function passed to DataFrameGroupBy.apply (undoing 2.)
4. after DataFrameGroupBy.apply (undoing 1.)

Roughly:

 def fn(df):
+    df = df.T
     …  # logic
-    return df
+    return df.T
 
 df2 = (
     df
+    .T
-    .groupby(lambda x:x, axis=1, level=0)
+    .groupby(lambda x:x, level=0)
     .apply(fn)
+    .T
 )

It would be nice to not have to do any of that, though!

[joshdempster]

Really unhappy with the decision to deprecate axis=1. I use it all the time for biological matrix data, and in fact handling axes agnostically is one of the ways pandas really shines over tidyverse for handling this data. The decision to move to a more verbose alternative is part of a really annoying trend in pandas towards the clunkiest possible syntax. Changing the behavior of .loc with missing labels is another prominent example; that one still irritates me every time I have to manually take the intersection of labels or rejigger things so .loc will work. And let's be real: the pandas API isn't exactly clean or streamlined now. It really doesn't need to demand even more verbosity from its users.

Hard disagree with Will. You personally may use pandas for columnar data 99% of the time. For me, about 80% of the time I'm working with matrices.

[joshdempster]

Also not really clear how 1) this can be the cause of so many bugs, and 2) users can just transpose their data. Either you can't always just transpose, or else there really isn't anything to debug.

[mmcky]

Love the work you all do on pandas -- we use it in our workflows everyday in a transformative way. I too am a bit disappointed this has been deprecated as the axis=1 was a nice unifying syntax that made for neater code and in my mind made it clear which axis operations like sum, groupby were acting on.

An example:

merged.groupby(level='Continent', axis=1).mean().plot()
plt.title('Average real minimum wage')
plt.ylabel('2015 USD')
plt.xlabel('Year')
plt.show()

is now

merged.T.groupby(level='Continent').mean().T.plot()
plt.title('Average real minimum wage')
plt.ylabel('2015 USD')
plt.xlabel('Year')
plt.show()

[helgeesch]

I'm also disappointed to see the axis attribute deprecated. Most of my use cases for groupby involve multi-index levels, whether on columns or rows. The axis attribute allowed seamless integration within wrapping methods and classes. Post-deprecation, I’ll need to implement something like:

def my_wrapping_method(df: pd.DataFrame, axis: int) -> pd.DataFrame:
    temp = df.T if axis == 1 else df
    
    # groupby, aggregation, manipulation to temp ...
    
    temp = df.T if axis == 1 else df
    return temp

not very elegant and unnecessary complexity

[oxygenbilly]

Our organization is using multi index with time series data. - like a columnar table
most of the cases actually depend on groupby(axis=1)
This is a very useful information - and in order to do so, we basically have to do dt.T.groupby().xxx.T. <- do two transpose which is not expressive at all