ENH: Reduce type requirements for the subset parameter in drop_duplicates/duplicated

[behrenhoff]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

I would like to pass a set to drop_duplicates like so:

df = get_some_df()
subset = {"column1", "column3"}
df_dropped = df.drop_duplicates(subset=subset)

According to type hints, subset is a Hashable | Sequence[Hashable] | None. The documentation says "column label or sequence of labels, optional".

The problem is, I would like to pass a set of columns. The name subset suggests that should be ok. And it does work indeed. However, a set is not a Sequence.

Can the requirements be lowered? Maybe to Collection? (or even Iterable, but that might come with problems).

Looking at the code: https://github.com/pandas-dev/pandas/blob/v2.2.2/pandas/core/frame.py#L6935

It checks if the subset is np.iterable, then a set is created and len(subset) is called. This could be done with collections as well.

Am I missing anything or could this Sequence be make a Collection?

Feature Description

If I am not mistaken, Sequence could simply be replaced with Collection in duplicated as well as drop_duplicates + in the docu.

Alternative Solutions

no alt solution required

Additional Context

No response

[kirill-bash]

Take

[rhshadrach]

This seems reasonable to me; cc @pandas-dev/pandas-core for any thoughts on the expansion of this parameter to sets. In the past, we've explicitly raised on sets for some arguments, but I believe in those cases it was because order mattered. Here, order does not matter.

[Dr-Irv]

I think this is a documentation and typing issue, not a code issue.

>>> df=pd.DataFrame.from_records([[1,2,3,4], [5,6,7,8], [10,20,3,4], [50,60,7,8]
], columns=["a", "b", "c", "d"])
>>> df
    a   b  c  d
0   1   2  3  4
1   5   6  7  8
2  10  20  3  4
3  50  60  7  8
>>> df.drop_duplicates(subset=["c", "d"])
   a  b  c  d
0  1  2  3  4
1  5  6  7  8
>>> df.drop_duplicates(subset=set(["c", "d"]))
   a  b  c  d
0  1  2  3  4
1  5  6  7  8

So you can pass a set today, but typing isn't allowing it if you are using pandas source to type check your code instead of pandas-stubs (which is the recommended way to type check user code)

I would suggest making a change here, but also in pandas-stubs at https://github.com/pandas-dev/pandas-stubs/blob/1846b34a44c41f151863fd5f9c20c4bc6a322d52/pandas-stubs/core/frame.pyi#L903 where we have yet to create a type for the subset argument.

[kirill-bash]

@Dr-Irv,

That is generally true except for single element sets.
My current PR should address that though.

>>> df=pd.DataFrame.from_records([[1,2,3,4], [5,6,7,8], [10,20,3,4], [50,60,7,8]
], columns=["a", "b", "c", "d"])
>>> df
    a   b  c  d
0   1   2  3  4
1   5   6  7  8
2  10  20  3  4
3  50  60  7  8
>>> df.drop_duplicates(subset=["c"])
   a  b  c  d
0  1  2  3  4
1  5  6  7  8
>>> df.drop_duplicates(subset=set(["c"]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/bb/bin/python/lib/python3.11/site-packages/pandas/core/frame.py", line 6802, in drop_duplicates
    result = self[-self.duplicated(subset, keep=keep)]
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/bb/bin/python/lib/python3.11/site-packages/pandas/core/frame.py", line 6938, in duplicated
    result = self[subset[0]].duplicated(keep)
                  ~~~~~~^^^
TypeError: 'set' object is not subscriptable

[Dr-Irv]

That is generally true except for single element sets. My current PR should address that though.

OK, that makes sense.