BUG: using NamedTuples with .loc works only sometimes

[AKuederle]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

# A df with two level MutltiIndex
df = pd.DataFrame(index=pd.MultiIndex.from_product([["A", "B"], ["a", "b", "c"]], names=["first", "second"]))

# Indexing with a normal tuple works as expected:
# normal subset
df.loc[("A", "b"), :]  # <- Works
# complicated subset
df.loc[("A", ["a", "b"]), :]  # <- Works

# Now the same with a named tuple
from collections import namedtuple

indexer_tuple = namedtuple("Indexer", df.index.names)

# simple subset with named tuple
df.loc[indexer_tuple(first="A", second="b")]  # <- Works

# complicated subset with named tuple
df.loc[indexer_tuple(first="A", second=["a", "b"]), :]  # <- DOES NOT WORK!
# Raises:
# InvalidIndexError: Indexer(first='A', second=['a', 'b'])

# However, converting back to a tuple works again
df.loc[tuple(indexer_tuple(first="A", second=["a", "b"])), :]  # <- works

Issue Description

When using loc with a namedtuple, only certain types of indexing work. That is surprising as I assumed that namedtuples (as they are a tuple subtype) would always just add a like a tuple.

The reason, why I used namedtuples in the first place was to make loc with multiple index levels easier to read. Therefore, it would be great, if namedtuples would work equivalently to tuples in loc

Expected Behavior

namedtuples should behave like normal tuples, as they are tuple sub-classes

Installed Versions

INSTALLED VERSIONS

commit : e8093ba
python : 3.8.12.final.0
python-bits : 64
OS : Linux
OS-release : 5.19.1-2-MANJARO
Version : #1 SMP PREEMPT_DYNAMIC Thu Aug 11 19:05:47 UTC 2022
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.4.3
numpy : 1.23.2
pytz : 2022.2.1
dateutil : 2.8.2
setuptools : 62.6.0
pip : 22.1.2
Cython : None
pytest : 7.1.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.1
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.4.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
markupsafe : 2.1.1
matplotlib : 3.5.3
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.9.0
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None

[jmoro0408]

Interestingly this seems to work fine until version 1.3.5. I wouldn't mind taking a better look into what changed between 1.3.5 and 1.4.0 to cause this behaviour if no one has any immediate ideas.

[AKuederle]

Thank you very much for investigating this!

…

On Thu, 18 Aug 2022, 23:16 James Moro, ***@***.***> wrote: Interestingly this seems to work fine until version 1.3.5. I wouldn't mind taking a better look into what changed between 1.3.5 and 1.4.0 if no one has any immediate ideas. — Reply to this email directly, view it on GitHub <#48124 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACSHBO5WMLXM4ODFDTW3UNTVZ2R3HANCNFSM56ZQLRLA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[tomaarsen]

@AKuederle
This behaviour can be reproduced on the main branch of pandas.

---

df.loc[("A", ["a", "b"]), :] 

and

df.loc[indexer_tuple(first="A", second=["a", "b"]), :]

will start to diverge here:

pandas/pandas/core/indexing.py

Line 1291 in 30efa5f

elif is_list_like_indexer(key):

This method explicitly checks if key is a list-like indexer that is not a NamedTuple. This is the implementation:

pandas/pandas/core/indexers/utils.py

Lines 67 to 80 in 30efa5f

	def is_list_like_indexer(key) -> bool:
	"""
	Check if we have a list-like indexer that is *not* a NamedTuple.
	Parameters
	----------
	key : object
	Returns
	-------
	bool
	"""
	# allow a list_like, but exclude NamedTuples which can be indexers
	return is_list_like(key) and not (isinstance(key, tuple) and type(key) is not tuple)

This causes the former to move to "nested tuple slicing", while the latter moves to "straight lookup". However, straight lookup fails, as Indexer(first='A', second=['a', 'b']) is clearly not a label present in the axis.

So, my question is, why are NamedTuples not allowed to be used in anything other than straight lookups? Perhaps it's simply an oversight, or perhaps there's some reason that I can't think of.
Changing is_list_like_indexer to is_list_like will cause all of your test cases to work perfectly.

[tomaarsen]

@jmoro0408

Interestingly this seems to work fine until version 1.3.5. I wouldn't mind taking a better look into what changed between 1.3.5 and 1.4.0 to cause this behaviour if no one has any immediate ideas.

In version v1.3.5, the following line still exists just before the aforementioned is_list_like_indexer:

pandas/pandas/core/indexing.py

Line 1138 in 66e3805

key = labels._get_partial_string_timestamp_match_key(key)

This line would convert Indexer(first='A', second=['a', 'b']) to ('A', ['a', 'b']). From that point onwards, both test cases I mentioned in my last comment will be the same, and both work correctly.

This method was deprecated in #42476, and this line was simply removed, which caused the bug.

[tomaarsen]

A similar bug exists with iloc and MultiIndex:

import pandas as pd
import numpy as np

# A df with two level MultiIndex
df = pd.DataFrame(np.arange(120).reshape(6, -1), index=pd.MultiIndex.from_product([["A", "B"], ["a", "b", "c"]], names=["first", "second"]))

# Indexing with a normal tuple works as expected:
# normal subset
print(df.iloc[1, 2])  # <- Works
# complicated subset
print(df.iloc[(1, [2, 3])])  # <- Works

# Now the same with a named tuple
from collections import namedtuple

indexer_tuple = namedtuple("Indexer", df.index.names)

print(df.iloc[tuple(indexer_tuple(first=1, second=[2, 3]))]) # <- Works
print(df.iloc[indexer_tuple(first=1, second=[2, 3])])  # <- DOES NOT WORK!

22
2    22
3    23
Name: (A, b), dtype: int64
2    22
3    23
Name: (A, b), dtype: int64
Traceback (most recent call last):
  File "/workspaces/pandas/demo.py", line 30, in <module>
  File "/workspaces/pandas/pandas/core/indexing.py", line 1074, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "/workspaces/pandas/pandas/core/indexing.py", line 1620, in _getitem_axis
    raise TypeError("Cannot index by location index with a non-integer key")
TypeError: Cannot index by location index with a non-integer key

This again originates from a _getitem_axis method, but now on _iLocIndexer instead of _LocIndexer.

There might be more cases where NamedTuples are not supported. Perhaps a better fix is to convert all NamedTuples to regular tuples at the start of a .loc, .iloc etc. call.

[phofl]

Converting is not an option, NamedTuples are valid Index Elements. Hence this could be a label based lookup.

is the iloc case also a regression?

[tomaarsen]

You're right, initial conversion would not work then.

I believe the iloc case was always broken; I know it is broken on main, v1.4.2 and v1.2.0.

[phofl]

Let’s open a different issue then with a broader discussion. I am a bit hesitant here, since I am not a fan of value dependent behavior.

the loc case worked accidentally on the 1.3.x branch

[tomaarsen]

the loc case worked accidentally on the 1.3.x branch

Indeed.

I'll open one

[phofl]

Iloc is probably easy, because it’s always positional, loc and regular getitem is tricky

[simonjayhawkins]

This method was deprecated in #42476, and this line was simply removed, which caused the bug.

indeed. from bisect, first bad commit: [9de158f] BUG: string slicing on MultiIndex DatetimeIndex level (#42476)