BUG: MultiIndex converts "\x00" to ""

[andreareina]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

l1 = ["", "\x00"]
i1 = pd.Index(l1)
print(i1.tolist()) # ['', '\x00']
print(i1.tolist() == l1) # True

l2 = [("",), ("\x00",)]
i2 = pd.Index(l2)
print(i2.tolist()) # [('',), ('',)]
print(i2.tolist() == l2) # False

Issue Description

Null-character strings in tuples are converted to the empty string when used as an index.

Expected Behavior

The null-character string should be preserved.

Installed Versions

INSTALLED VERSIONS

commit : 4bfe3d0
python : 3.8.11.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.0-104-generic
Version : #118~18.04.1-Ubuntu SMP Thu Mar 3 13:53:15 UTC 2022
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_SG.UTF-8
LOCALE : en_SG.UTF-8

pandas : 1.4.2
numpy : 1.22.3
pytz : 2022.1
dateutil : 2.8.2
pip : 21.2.3
setuptools : 57.4.0
Cython : None
pytest : 5.4.3
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
markupsafe : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None

[simonjayhawkins]

Thanks @andreareina for the report. Can you give the issue an informative title.

Null-character strings in tuples are converted to the empty string when used as an index.

Tuples used in the Index constructor create a single level MultiIndex. Under the hood the MultiIndex values are stored as factorized codes so this could be where the discrepancy occurs. Further investigation and PRs welcome.

[andreareina]

Seems to be happening in _libs.hashtable.StringHashTable but I'm unable to follow the cython. Probably the underlying C(++?) expecting null-terminated strings.

Code excerpted (and modified to replace return values from dependencies) from core.algorithms.factorize_array()

pandas/pandas/core/algorithms.py

Lines 524 to 565 in 4bfe3d0

	def factorize_array(
	values: np.ndarray,
	na_sentinel: int = -1,
	size_hint: int | None = None,
	na_value=None,
	mask: np.ndarray | None = None,
	) -> tuple[npt.NDArray[np.intp], np.ndarray]:
	"""
	Factorize a numpy array to codes and uniques.
	This doesn't do any coercion of types or unboxing before factorization.
	Parameters
	----------
	values : ndarray
	na_sentinel : int, default -1
	size_hint : int, optional
	Passed through to the hashtable's 'get_labels' method
	na_value : object, optional
	A value in `values` to consider missing. Note: only use this
	parameter when you know that you don't have any values pandas would
	consider missing in the array (NaN for float data, iNaT for
	datetimes, etc.).
	mask : ndarray[bool], optional
	If not None, the mask is used as indicator for missing values
	(True = missing, False = valid) instead of `na_value` or
	condition "val != val".
	Returns
	-------
	codes : ndarray[np.intp]
	uniques : ndarray
	"""
	hash_klass, values = _get_data_algo(values)
	table = hash_klass(size_hint or len(values))
	uniques, codes = table.factorize(
	values, na_sentinel=na_sentinel, na_value=na_value, mask=mask
	)
	codes = ensure_platform_int(codes)
	return codes, uniques

import numpy as np
import pandas as pd

values = np.array(["", "\x00"], dtype=object)
size_hint = None
na_sentinel = -1
na_value = None
mask = None

# Excerpted from https://github.com/pandas-dev/pandas/blob/4bfe3d07b4858144c219b9346329027024102ab6/pandas/core/algorithms.py#L524-L565 (factorize_array)

# hash_klass, values = _get_data_algo(values)
hash_klass = pd._libs.hashtable.StringHashTable

table = hash_klass(size_hint or len(values))
uniques, codes = table.factorize(
    values, na_sentinel=na_sentinel, na_value=na_value, mask=mask
)
print(repr(uniques)) #=> array([''], dtype=object)

# codes = ensure_platform_int(codes)

[mzeitlin11]

Probably related to #34551?

[simonjayhawkins]

Probably related to #34551?

appears so. there is also a comment on that issue, #34551 (comment) regarding MultiIndex and comments regarding other methods that use the factorize algo for Null-character strings.

can probably close this as duplicate.

[mroeschke]

Closing as a duplicate of #34551