BUG: df.groupby(axis=1).size() returns weird output

[vnlitvinov]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
arrays = [['Falcon', 'Falcon', 'Parrot', 'Parrot'], ['Captive', 'Wild', 'Captive', 'Wild']]
index = pd.MultiIndex.from_arrays(arrays, names=('Animal', 'Type'))
df = pd.DataFrame({'Max Speed': [390., 350., 30., 20.], 'stuff':[1, 2, 3, 4]}, index=index)
print(f'original df:\n{df}\n\n')
print(f'groupby1:\n{df.groupby(level=0,axis=1).size()}\n\n')
print(f'groupby2:\n{df.T.groupby(level=1,axis=1).size()}\n\n')

# simpler case
df = pd.DataFrame(
        {
            "col1": [0, 3, 2, 3],
            "col2": [4, 1, 6, 7],
            "col3": [3, 8, 2, 10],
            "col4": [1, 13, 6, 15],
            "col5": [-4, 5, 6, -7],
        }
    )
print(f'simple groupby:\n{df.groupby(axis=1, by=[1, 2, 3, 2, 1]).size()}')

Issue Description

In the example, I think that starting on 1.5, .size() is providing multi-column DataFrame (instead of Series) with all columns equal to each other (and names identical to original df columns).

I wonder if that change is intentional, I sure hope it's not.

Output of the reproducer on pandas 1.5

original df:
                Max Speed  stuff
Animal Type
Falcon Captive      390.0      1
       Wild         350.0      2
Parrot Captive       30.0      3
       Wild          20.0      4


groupby1:
                Max Speed  stuff
Animal Type
Falcon Captive          1      1
       Wild             1      1
Parrot Captive          1      1
       Wild             1      1


groupby2:
Type       Captive  Wild
Max Speed        2     2
stuff            2     2


simple groupby:
   1  2  3
0  2  2  1
1  2  2  1
2  2  2  1
3  2  2  1

Expected Behavior

Expected output (on pandas 1.4.4)

original df:
                Max Speed  stuff
Animal Type
Falcon Captive      390.0      1
       Wild         350.0      2
Parrot Captive       30.0      3
       Wild          20.0      4


groupby1:
Max Speed    1
stuff        1
dtype: int64


groupby2:
Type
Captive    2
Wild       2
dtype: int64


simple groupby:
1    2
2    2
3    1
dtype: int64

Installed Versions

pd.show_versions()

INSTALLED VERSIONS
------------------
commit           : 87cfe4e38bafe7300a6003a1d18bd80f3f77c763
python           : 3.8.13.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
Version          : 10.0.22000
machine          : AMD64
processor        : AMD64 Family 25 Model 80 Stepping 0, AuthenticAMD
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : Russian_Russia.1251

pandas           : 1.5.0
numpy            : 1.23.3
pytz             : 2022.2.1
dateutil         : 2.8.2
setuptools       : 65.3.0
pip              : 22.2.2
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
brotli           : None
fastparquet      : None
fsspec           : None
gcsfs            : None
matplotlib       : None
numba            : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pyreadstat       : None
pyxlsb           : None
s3fs             : None
scipy            : None
snappy           : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
zstandard        : None
tzdata           : None

[vnlitvinov]

I've found a simpler reproducer which does not include multi-index but exhibits same weird output for .size(), so I've added that, too.

[phofl]

Hi, thanks for your report.

commit 15a06d3d9e7656afff239da7a295a7b684456680
Author: Richard Shadrach <[email protected]>
Date:   Mon Feb 14 20:00:54 2022 -0500

    BUG: groupby.size and groupby.transform('size') incorrect for axis=1 (#45987)

cc @rhshadrach

[rhshadrach]

Thanks for the report! I believe in #45987 I got caught up with making size behave like other reducers. Consider your example with count instead:

   1  2  3
0  2  2  1
1  2  2  1
2  2  2  1
3  2  2  1

Unlike size, count isn't guaranteed to be constant for every row (nulls aren't counted), so we need to output in this format. However, with axis=0 size returns a Series unlike count as well, and I think this behavior is useful even though it breaks the pattern of a typical reducer. I think we should revert this change.