REGR: fillna not filling NaNs after pivot without explicitly listing pivot values

[miles82]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Hi,

I have this (simplified) piece of code which seems to have stopped working correctly in Pandas 1.1:

import pandas as pd

data = [
    [1, 1, 1, 1.],
    [2, 2, 2, 2.],
    [3, 3, 3, 3.],
]

df = pd.DataFrame(data, columns=['i1', 'i2', 'i3', 'f1'])
df = df.pivot('i1', 'i2')
df = df.fillna(0)
print(df)

Problem description

This is the output in Pandas 1.1.2:

     i3             f1
i2    1    2    3    1    2    3
i1
1   1.0  NaN  NaN  1.0  0.0  0.0
2   NaN  2.0  NaN  0.0  2.0  0.0
3   NaN  NaN  3.0  0.0  0.0  3.0

It contains NaNs after using fillna. It can be fixed by explicitly providing remaining columns as values to pivot (which should be the default):
df = df.pivot('i1', 'i2', values=['i3', 'f1'])

It also works if all columns are of the same type, like when I change f1 to contain ints instead of floats. In that case I don't need to specify the values.

Expected Output

This is the output on Pandas 1.0.5 and 0.25.3:

     i3             f1
i2    1    2    3    1    2    3
i1
1   1.0  0.0  0.0  1.0  0.0  0.0
2   0.0  2.0  0.0  0.0  2.0  0.0
3   0.0  0.0  3.0  0.0  0.0  3.0

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 2a7d332
python : 3.7.8.final.0
python-bits : 32
OS : Windows
OS-release : 10
Version : 10.0.18362
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 9, GenuineIntel
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : None.None

pandas : 1.1.2
numpy : 1.19.2
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.3
setuptools : 50.3.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.5.2
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.18.1
pandas_datareader: None
bs4 : 4.9.1
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.5.2
sqlalchemy : 1.3.18
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None
None

[samilAyoub]

I confirm this issue !

[dsaxton]

Thanks @miles82 for the nice report and @samilAyoub for confirming.

760ba37 is the first bad commit
commit 760ba37
Author: jbrockmendel [email protected]
Date: Tue May 26 15:47:18 2020 -0700

CLN: _consolidate_inplace less (#34389)

pandas/core/generic.py | 9 ---------
pandas/core/internals/managers.py | 10 ----------
2 files changed, 19 deletions(-)

PR #34389
cc @jbrockmendel

[simonjayhawkins]

moved off 1.1.3 milestone, see #36668 (comment)

[simonjayhawkins]

moved off 1.1.4 milestone (scheduled for release tomorrow)

[jorisvandenbossche]

@jbrockmendel do you have time to take a look at this?