BUG: concat losing column dtype for extension arrays and object dtype

[phofl]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

df1 = pd.DataFrame([[0, 1, 1]], columns=pd.Index([1, 2, 3], dtype="Int64"))
df2 = pd.DataFrame([[0, 1]], columns=pd.Index([1, 2], dtype="Int64"))
result = pd.concat([df1, df2], ignore_index=True, join="outer", sort=True)
expected = pd.DataFrame([[0, 1, 1.0], [0, 1, np.nan]], columns=pd.Index([1, 2, 3], dtype="Int64"))
tm.assert_frame_equal(result, expected)

Issue Description

this raises

Expected Behavior

The above code should not return an error

Installed Versions

INSTALLED VERSIONS ------------------ commit : https://github.com/pandas-dev/pandas/commit/5465f54f7640cbb4be9b223846823d4935142431 python : 3.8.12.final.0 python-bits : 64 OS : Linux OS-release : 5.13.0-48-generic Version : https://github.com//issues/54~20.04.1-Ubuntu SMP Thu Jun 2 23:37:17 UTC 2022 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 1.5.0.dev0+939.g5465f54f76.dirty
numpy : 1.22.0
pytz : 2021.1
dateutil : 2.8.2
setuptools : 59.8.0
pip : 21.3.1
Cython : 0.29.30
pytest : 7.1.1
hypothesis : 6.23.1
sphinx : 4.2.0
blosc : None
feather : None
xlsxwriter : 3.0.1
lxml.etree : 4.6.3
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.0.1
IPython : 7.28.0
pandas_datareader: None
bs4 : 4.10.0
bottleneck : 1.3.2
brotli :
fastparquet : 0.7.1
fsspec : 2021.11.0
gcsfs : 2021.05.0
matplotlib : 3.4.3
numba : 0.53.1
numexpr : 2.7.3
odfpy : None
openpyxl : 3.0.9
pandas_gbq : None
pyarrow : 5.0.0
pyreadstat : 1.1.2
pyxlsb : None
s3fs : 2021.11.0
scipy : 1.7.2
snappy :
sqlalchemy : 1.4.25
tables : 3.6.1
tabulate : 0.8.9
xarray : 0.18.0
xlrd : 2.0.1
xlwt : 1.3.0
zstandard : None

[phofl]

This is still true after #47330

[simonjayhawkins]

extension arrays in the index was an enhancement in 1.4 so any fix could be considered as a backport candidate.

[phofl]

This is a bit tricky, because it depends on #47330, which is not really a regression, meaning this would have side effects

[simonjayhawkins]

because _get_combined_index is now gaining the original dtype explicitly?

[phofl]

Yeah, _get_combined_index is preserving the dtype of the incoming indexes but it does not on 1.4.3. The metadata preservation has side effects into other functions, like assert_index_equal.

[simonjayhawkins]

the change to _get_combined_index to construct the Index instead of using .sort_values() was done in #47206 (i.e. not yet released), so changing that further is probably not an issue?

As mentioned in #47325 (comment), does that fix now have the rogue warning and also does it work for Indexes with duplicates (not checked just noticed that it looks like there maybe some constraints to using safe_sort in place of .sort_values())

[phofl]

The dtype is only kept after #47330, which fixes a bug not a regression, hence the side effects comment.

The fix does not have a warning, have to check with duplicates

[phofl]

We don't pass codes, so this is not an issue in our case

[simonjayhawkins]

This is a bit tricky, because it depends on #47330, which is not really a regression, meaning this would have side effects

ok will review any fix for this once #47330 is merged and reassess. moving to 1.4.4 for now.

[simonjayhawkins]

milestone changed. see #47586 (comment)

[simonjayhawkins]

example of this occuring for object dtype

df1 = pd.DataFrame([[0, 1, 1]], columns=pd.Index([1, 2, 3], dtype="object"))
df2 = pd.DataFrame([[0, 1]], columns=pd.Index([1, 2], dtype="object"))
result = pd.concat([df1, df2], ignore_index=True, join="outer", sort=True)
print(result.columns)
# Int64Index([1, 2, 3], dtype='int64')