BUG: If aggregation function returns NaN the order of the index on the resulting df is not maintained

[mfcabrera]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

df = pd.DataFrame(
             [
                 ("1", 1, 0, 0),
                 ("2", 2, 0, 0),
                 ("3", 3, 0, 0),
                 ("4", 4, 5, 4),
                 ("5", 5, 6, 6),
                 ("6", 6, 7, 7),
             ],
             columns=("item", "att1", "att2", "att3"),
    )

def myagg1(s):
    return s.sum()

def myagg2(s):
       return s.sum() / 2
 
aggs1 = ["sum", myagg1, "count", "min"]    
aggs2 = ["sum", myagg1, "count", "min"]    

df.agg(aggs1) # order of the index is maintained
#          item  att1  att2  att3
# sum     123456    21    18    17
# myagg1  123456    21    18    17
# count        6     6     6     6
# min          1     1     0     0

df.agg(aggs2) # myagg2 returns nan and the result is sent to the last row
#           item  att1  att2  att3
# sum     123456  21.0  18.0  17.0
# count        6   6.0   6.0   6.0
# min          1   1.0   0.0   0.0
# myagg2     NaN  10.5   9.0   8.5

Problem description

the order of the index on the resulting dataframe should not affected by the result of one of the aggregation functions.

Expected Output

df.agg(aggs2) # Regardless of the result the order is maintained
#           item  att1  att2  att3
# sum     123456  21.0  18.0  17.0
# myagg2     NaN  10.5   9.0   8.5
# count        6   6.0   6.0   6.0
# min          1   1.0   0.0   0.0

Output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.8.1.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 19.0.0
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : None
LANG             : de_DE.UTF-8
LOCALE           : de_DE.UTF-8

pandas           : 1.0.3
numpy            : 1.18.1
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 20.0.2
setuptools       : 45.2.0.post20200210
Cython           : None
pytest           : 5.4.1
hypothesis       : 5.8.0
sphinx           : 2.4.4
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.11.1
IPython          : 7.12.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pytest           : 5.4.1
pyxlsb           : None
s3fs             : None
scipy            : 1.4.1
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
numba            : None

[TomAugspurger]

Thanks for the report @mfcabrera. Are you interested in working on this?

[DriesSchaumont]

take

[DriesSchaumont]

Seems to be a problem with the arguments passed to concat:

ser1 = pd.Series(["1","3"], index=pd.Index(["a", "c"], dtype="object"), dtype="object", name="test1")
ser2 = pd.Series([3, 4, 5], index=pd.Index(["a", "b", "c"], dtype="object"), dtype="float64", name="test2")
ser3 = pd.Series([6,7,8], index=pd.Index(["a", "b", "c"], dtype="object"), dtype="float64", name="test3")
pd.concat([ser1, ser2, ser3], axis=1, sort=False, keys=["test1", "test2", "test3"])
  test1  test2  test3
a     1    3.0    6.0
c     3    5.0    8.0
b   NaN    4.0    7.0