ENH: .isin() method should use __contains__ rather than __iter__ for user-defined classes to determine presence.

[f3ss1]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

Right now, if you would define a user class:

class MyClass:
    def __init__(self):
        self.collection = [1, 2, 3]
        self.another_collection = [4, 5, 6]
    
    def __contains__(self, item):
        return item in self.collection
    
    def __iter__(self):
        yield from self.another_collection

and would then initialize a pandas dataframe like this:

example_dataframe = pd.DataFrame(
    {
        'column_name': [3, 1, 4, 6, 13],
        'another_column_name': ['tolly', 'trolly', 'telly', 'belly', 'nelly']
    }
)

and would then call the .isin() method like this:

class_instance = MyClass()
example_dataframe['column_name'].isin(class_instance)

you would actually get this output:

False
False
True
True
False

which is if the values from self.another_collections specified in __iter__ are checked, rather than self.collection from __contains__. I do realize that this might stem from compatibility with other libraries, but this seems counter-intuitive.

Feature Description

A solution I suggest is either to change the behavior (which might result into ruining some peoples code, I believe), or adding a flag (which would lead to more complexity, I guess).

Alternative Solutions

See above.

Additional Context

No response

[rhshadrach]

isin uses a hash table for performance - this would not work with __contains__. Is there a real-world example where the results would differ?

I'd be okay with leaking a bit of the implementation details in the docstring - that isin iterates through the values in the passed list-like.

[f3ss1]

I believe the real-life example I have might not be the best programming practice. In my scenario, I have a dictionary that maps item names to item instances, and the __contains__ method works for both class names and instances. However, I'm starting to doubt if this is a good design choice in fact.

Still, I think that the current isin implementation and its documentation need some adjustments as they seem a bit misleading. Although it mentions that the object should be Iterable, implying that __contains__ might not be used, it also suggests using a dict, which contradicts this. Maybe introducing how it works with an Attention section in the documentations is indeed the most effective solution.

[rhshadrach]

Although it mentions that the object should be Iterable, implying that __contains__ might not be used, it also suggests using a dict, which contradicts this.

Can you elaborate what the contradiction is?

[f3ss1]

I believe my comment might be misleading :) It's not the contradiction in general for Python (of course dict is iterable as well) rather than in my personal opinion: when I see a dict, the first thing I think about is hashing and doing stuff like my_dict["a"] which leads to thinking that it would use __contains__.

[rhshadrach]

I'm good with a note that we use __iter__ over __contains__ in a notes section of the docstring.

[AnaDenisa]

take

[Ezibenroc]

Hello,

Is there a real-world example where the results would differ?

I think there is yes: bloom filters.
I would like to filter a pandas dataframe with a bloom filter, but stumbled on this issue:

TypeError: only list-like objects are allowed to be passed to isin(), you passed a `Bloom`

The problem is that you cannot iterate on such data structures, it doesn't make sense. The only thing you can do is test the membership of an element.

[rhshadrach]

@Ezibenroc - are you able to use df[df["a"].map(lambda x: x in bloom_filter)]?

[Ezibenroc]

Good point, it is slightly less readable, but it should work well, thank you!

[rhshadrach]

My main opposition to supporting this in isin is that it would require making isin use a non-vectorized approach, which makes the semantics more confusing.