BUG: "None" in column name tuple changed to "nan" after concat

[make-ing]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

df_1=pd.DataFrame({("column A", 1):[1,2,3]})
print(df_1)
df_2=pd.DataFrame({("column B", None):[4,5,6]})
print(df_2)
new_df = pd.concat([df_1, df_2], axis=1)
print(new_df)

Issue Description

After concatenating the two dataframes the second field of the second column name tuple changed from None to nan. I'm not sure if this is the desired behaviour or a bug.

Expected Behavior

I expect the column name of the concatenated series to be exactly the same as the name of the series: ("column B", None) and not ("column B", nan)

Installed Versions

INSTALLED VERSIONS

commit : a671b5a
python : 3.12.0.final.0
python-bits : 64
OS : Linux
OS-release : 6.5.0-14-generic
Version : #14~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov 20 18:15:30 UTC 2
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.1.4
numpy : 1.26.3
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : None
pip : 23.2.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader : None
bs4 : None
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.4
qtpy : None
pyqt5 : None

[rhshadrach]

@jorisvandenbossche @phofl - I think this would be covered by the Ice Cream Agreement, but haven't checked if there are duplicate reports about NA values in the labels of an index.

[make-ing]

sorry for asking, but what is the "Ice Cream Agreement"?

[jorisvandenbossche]

After concatenating the two dataframes the second field of the second column name tuple changed from None to nan. I'm not sure if this is the desired behaviour or a bug.

That's not the behaviour that I see with pandas 2.1.4. For me, this is already NaN before the concatenation. The printed output from running your example:

  column A
         1
0        1
1        2
2        3
  column B
       NaN
0        4
1        5
2        6
  column A column B
         1      NaN
0        1        4
1        2        5
2        3        6

So it is the dataframe construction with pd.DataFrame({("column B", None): [4,5,6]}) that already creates this "NaN". But, in this case of a MultiIndex columns, this None is just translated to a missing value in the MultiIndex (under the hood, this is stored as an integer code of -1), and that is printed as NaN:

In [4]: df_2.columns
Out[4]: 
MultiIndex([('column B', nan)])

In [5]: df_2.columns.levels
Out[5]: FrozenList([['column B'], []])

In [6]: df_2.columns.codes
Out[6]: FrozenList([[0], [-1]])

So @rhshadrach I don't think this is related to the NA discussions (just that if NA would become the default, this would be used as the missing value sentinel for MultiIndex as well).

sorry for asking, but what is the "Ice Cream Agreement"?

No problem for asking ;), that's referring to an in-person discussion about missing value support we had some months ago, for which we haven't yet made an official proposal or name.

[rhshadrach]

Thanks @jorisvandenbossche for determining the issue is not with concat.

So @rhshadrach I don't think this is related to the NA discussions (just that if NA would become the default, this would be used as the missing value sentinel for MultiIndex as well).

My guess is that a user might be surprised that pd.DataFrame({("column B", None):[4,5,6]}) resulted in NaN as the column label. Whether it happens during construction or during concat, it seems to me the change from None to NaN would be resolved by presenting and treating both as NA.

[make-ing]

That's not the behaviour that I see with pandas 2.1.4. For me, this is already NaN before the concatenation.

Thanks for pointing this out... i'm not sure why i missed it. I can confirm this also on pandas 2.2.0.

In newer versions it also seems ok since i can get the column by either requesting the MutliIndex with None or with np.nan as the second index field.

I will close this issue since its not really a bug and at least in newer versions of pandas i can still get the column with None in the MultiIndex.

Thanks for all your hard work you do on pandas and other open source projects.