DataFrame.drop with empty DatetimeIndex behaves differently than passing empty Index

[katsuya-horiuchi]

Code Sample, a copy-pastable example if possible

import pandas as pd

df = pd.DataFrame({
    {'value': [1, 2]},
    index=[pd.Timestamp('2019-01-01'), pd.Timestamp('2019-01-01')]
})
df = df.drop(df[df['value'] > 10].index)

Problem description

When calling drop for dataframe with DatetimeIndex, the function may raise an exception. See the following traceback:

Traceback (most recent call last):
  File "main.py", line 7, in <module>
    df = df.drop(df[df['value'] > 10].index)
  File "pandas/pandas/core/frame.py", line 4042, in drop
    errors=errors,
  File "pandas/pandas/core/generic.py", line 3891, in drop
    obj = obj._drop_axis(labels, axis, level=level, errors=errors)
  File "pandas/pandas/core/generic.py", line 3940, in _drop_axis
    labels_missing = (axis.get_indexer_for(labels) == -1).any()
  File "pandas/pandas/core/indexes/base.py", line 4769, in get_indexer_for
    indexer, _ = self.get_indexer_non_unique(target, **kwargs)
  File "pandas/pandas/core/indexes/base.py", line 4752, in get_indexer_non_unique
    indexer, missing = self._engine.get_indexer_non_unique(tgt_values)
  File "pandas/_libs/index.pyx", line 291, in pandas._libs.index.IndexEngine.get_indexer_non_unique
    stargets = set(targets)
TypeError: 'NoneType' object is not iterable

This can be avoided by not using DataFrame.drop like this:

 df = df[df['value'] <= 10]

But nonetheless I'm reporting the issue as I expect drop to not raise the exception.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 9f93d57
python : 3.7.4.final.0
python-bits : 64
OS : Linux
OS-release : 4.19.0-5-amd64
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 0.25.0.dev0+1184.g9f93d5702
numpy : 1.16.2
pytz : 2019.2
dateutil : 2.8.0
pip : 19.2.2
setuptools : 41.1.0
Cython : 0.29.13
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None

[mroeschke]

Your example is equivalent to df.drop([]). Our documentation states that we should be raising a KeyError instead in this case since [] in not found in the axis : https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.drop.html

[katsuya-horiuchi]

Thank you @mroeschke

My initial thought was the issue is caused by duplicated values in index (i.e. pd.Timestamp('2019-01-01')).
However, the following will work despite duplication in the index:

df = pd.DataFrame(
    {'value': [1, 2]},
    index=[0, 0]
)
df = df.drop(df[df['value'] > 10].index)

So I'm wondering if this difference was intentional.

[mroeschke]

Ah I see. Thanks for the counter example. I agree then I would expected passing an empty DatetimeIndex to operate the same as passing an empty Index.

[katsuya-horiuchi]

Interestingly, I got TypeError while trying df.drop([]). The following is the traceback.

Traceback (most recent call last):
  File "main.py", line 10, in <module>
    print(df.drop([]))
  File "pandas/pandas/core/frame.py", line 4042, in drop
    errors=errors,
  File "pandas/pandas/core/generic.py", line 3893, in drop
    obj = obj._drop_axis(labels, axis, level=level, errors=errors)
  File "pandas/pandas/core/generic.py", line 3942, in _drop_axis
    labels_missing = (axis.get_indexer_for(labels) == -1).any()
  File "pandas/pandas/core/indexes/base.py", line 4769, in get_indexer_for
    indexer, _ = self.get_indexer_non_unique(target, **kwargs)
  File "pandas/pandas/core/indexes/base.py", line 4752, in get_indexer_non_unique
    indexer, missing = self._engine.get_indexer_non_unique(tgt_values)
  File "pandas/_libs/index.pyx", line 291, in pandas._libs.index.IndexEngine.get_indexer_non_unique
    stargets = set(targets)
TypeError: 'NoneType' object is not iterable

With that said, would it make sense to not raise these exceptions when passing empty list/index to DataFrame.drop()?

If so, I might be able to contribute.
The change I'm thinking would align with my expectation: df = df.drop(df[df['value'] > 10].index) would behave the same way as df = df[df['value'] <= 10]

[mroeschke]

Agreed with your conclusion, I think DataFrame.drop([]) should just be a no-op like you demonstrated in #27994 (comment)

[mroeschke]

This looks to work on master now. Could use a test

In [9]: df = pd.DataFrame(
   ...:     {'value': [1, 2]},
   ...:     index=[pd.Timestamp('2019-01-01'), pd.Timestamp('2019-01-01')]
   ...: )

In [10]: df = df.drop(df[df['value'] > 10].index)
In [12]: df
Out[12]:
            value
2019-01-01      1
2019-01-01      2

[ghost]

@mroeschke I'm looking for an issue to pickup for my first contribution, was wondering if I could take this up.

[mroeschke]

Yes go for it @mikephung122.

[ghost]

@mroeschke I submitted a Pull Request (#42746) for this issue, could you take a look when you get the chance?

[phofl]

fixed in #42746