Skip to content

BUG: drop_duplicates() doesn't drop zero-column rows #40257

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
3 tasks done
lealanko-rt opened this issue Mar 5, 2021 · 1 comment
Closed
3 tasks done

BUG: drop_duplicates() doesn't drop zero-column rows #40257

lealanko-rt opened this issue Mar 5, 2021 · 1 comment
Labels
Bug Duplicate Report Duplicate issue or pull request duplicated duplicated, drop_duplicates

Comments

@lealanko-rt
Copy link

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • (optional) I have confirmed this bug exists on the master branch of pandas.


Code Sample, a copy-pastable example

import pandas as pd
df_01 = pd.DataFrame([[]])
df_02 = pd.DataFrame([[],[]])
assert df_02.drop_duplicates().equals(df_01)
assert df_02.duplicated().equals(pd.Series([False,True]))

Problem description

Duplicates checking fails on various zero-column dataframes. They check for self.empty for a fast-path bypass, when they really should be checking e.g. self.index.empty.

Expected Output

The above assertions should succeed.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 7d32926
python : 3.7.3.final.0
python-bits : 64
OS : Linux
OS-release : 4.4.0-17763-Microsoft
Version : #864-Microsoft Thu Nov 07 15:22:00 PST 2019
machine : x86_64
processor :
byteorder : little
LC_ALL : C
LANG : en_US.UTF-8
LOCALE : None.None

pandas : 1.2.2
numpy : 1.20.1
pytz : 2021.1
dateutil : 2.8.1
pip : 18.1
setuptools : 40.8.0
Cython : None
pytest : None
hypothesis : None
sphinx : 3.4.3
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.4.2
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 5.8.0
pandas_datareader: None
bs4 : 4.7.1
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.0.2
numexpr : 2.6.9
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.1.0
sqlalchemy : None
tables : 3.4.4
tabulate : 0.8.2
xarray : None
xlrd : None
xlwt : None
numba : None

@lealanko-rt lealanko-rt added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Mar 5, 2021
@mroeschke mroeschke added duplicated duplicated, drop_duplicates and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Aug 19, 2021
@simonjayhawkins
Copy link
Member

Thanks @lealanko-rt for the report. This is a duplicate of #12869. There is also some discussion there on what the result should be. closing this to help keep discussion in one place.

@simonjayhawkins simonjayhawkins added the Duplicate Report Duplicate issue or pull request label Jun 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Duplicate Report Duplicate issue or pull request duplicated duplicated, drop_duplicates
Projects
None yet
Development

No branches or pull requests

3 participants