MultiIndexHashEngine can't manage mixed type tuples

[toobaz]

Code Sample, a copy-pastable example if possible

In [2]: mi = pd.MultiIndex.from_product([[True, False], range(1, 10)])

In [3]: mi.get_loc((False, 1))
Out[3]: 9

In [4]: mi = pd.MultiIndex.from_product([[True, False], range(1, 10000)])

In [5]: mi.get_loc((False, 1))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-68f8698878ec> in <module>()
----> 1 mi.get_loc((False, 1))

/home/pietro/nobackup/repo/pandas/pandas/core/indexes/multi.py in get_loc(self, key, method)
   2134             key = _values_from_object(key)
   2135             key = tuple(map(_maybe_str_to_time_stamp, key, self.levels))
-> 2136             return self._engine.get_loc(key)
   2137 
   2138         # -- partial selection or non-unique index

/home/pietro/nobackup/repo/pandas/pandas/_libs/index.pyx in pandas._libs.index.MultiIndexHashEngine.get_loc (pandas/_libs/index.c:15854)()

/home/pietro/nobackup/repo/pandas/pandas/_libs/index.pyx in pandas._libs.index.MultiIndexHashEngine.get_loc (pandas/_libs/index.c:15701)()

/home/pietro/nobackup/repo/pandas/pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.MultiIndexHashTable.get_item (pandas/_libs/hashtable.c:24621)()

/home/pietro/nobackup/repo/pandas/pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.MultiIndexHashTable.get_item (pandas/_libs/hashtable.c:24468)()

/home/pietro/nobackup/repo/pandas/pandas/core/indexes/multi.py in _hashed_indexing_key(self, key)
    819         key = tuple([f(k, stringify)
    820                      for k, stringify in zip(key, self._have_mixed_levels)])
--> 821         return hash_tuple(key)
    822 
    823     @Appender(base._shared_docs['duplicated'] % _index_doc_kwargs)

/home/pietro/nobackup/repo/pandas/pandas/core/util/hashing.py in hash_tuple(val, encoding, hash_key)
    186               for v in val)
    187 
--> 188     h = _combine_hash_arrays(hashes, len(val))[0]
    189 
    190     return h

/home/pietro/nobackup/repo/pandas/pandas/core/util/hashing.py in _combine_hash_arrays(arrays, num_items)
     31     """
     32     try:
---> 33         first = next(arrays)
     34     except StopIteration:
     35         return np.array([], dtype=np.uint64)

/home/pietro/nobackup/repo/pandas/pandas/core/util/hashing.py in <genexpr>(.0)
    184     """
    185     hashes = (_hash_scalar(v, encoding=encoding, hash_key=hash_key)
--> 186               for v in val)
    187 
    188     h = _combine_hash_arrays(hashes, len(val))[0]

/home/pietro/nobackup/repo/pandas/pandas/core/util/hashing.py in _hash_scalar(val, encoding, hash_key)
    330 
    331     return hash_array(vals, hash_key=hash_key, encoding=encoding,
--> 332                       categorize=False)

/home/pietro/nobackup/repo/pandas/pandas/core/util/hashing.py in hash_array(vals, encoding, hash_key, categorize)
    290 
    291         try:
--> 292             vals = hashing.hash_object_array(vals, hash_key, encoding)
    293         except TypeError:
    294             # we have mixed types

/home/pietro/nobackup/repo/pandas/pandas/_libs/hashing.pyx in pandas._libs.hashing.hash_object_array (pandas/_libs/hashing.c:1764)()

ValueError: Does not understand character buffer dtype format string ('?')

In [6]: mi = pd.MultiIndex.from_product([[1, 0], range(1, 10000)])

In [7]: mi.get_loc((1, 1))
Out[7]: 0

Problem description

The two engines should give the same result.

Expected Output

Out[3]:

Output of pd.show_versions()

INSTALLED VERSIONS

commit: f745e52
python: 3.5.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.0-3-amd64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: it_IT.UTF-8
LOCALE: it_IT.UTF-8

pandas: 0.22.0.dev0+241.gf745e52e1.dirty
pytest: 3.2.3
pip: 9.0.1
setuptools: 36.7.0
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.19.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: 1.5.6
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.0dev
tables: 3.3.0
numexpr: 2.6.1
feather: 0.3.1
matplotlib: 2.0.0
openpyxl: None
xlrd: 1.0.0
xlwt: 1.1.2
xlsxwriter: 0.9.6
lxml: None
bs4: 4.5.3
html5lib: 0.999999999
sqlalchemy: 1.0.15
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: 0.2.1

[gfyoung]

Complex dtypes are probably not pandas cup of tea. That being said, we should still try to make our hashtables more robust. Have a look around and see what you can find.

[jreback]

@gfyoung
complex dtypes are pandas cup of tea, by-definition a tuple IS mixed dtype. so this is for sure in scope.

[gfyoung]

complex dtypes are pandas cup of tea, by-definition a tuple IS mixed dtype

Okay, good to know. I was under a different impression as you can tell.

[toobaz]

Well, I guess that in this specific case, the fact that the tuple is "complex" or not should be irrelevant, because it is just a container for values that refer to different levels - and the levels are homogeneous and nice, and the values are of the desired dtypes. So yes, this should be pretty standard.

(As opposed, for instance, to a tuple as a key for a flat index - which, ironically, works just fine)