BUG: equal-valued series should work with qcut

[memeplex]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

s = pd.Series([1] * 5)
pd.qcut(s, 3, duplicates='drop')


=>

0 NaN
1 NaN
2 NaN
3 NaN
4 NaN
dtype: category
Categories (0, interval[float64, right]): []

Issue Description

The series in the example fails to produce a partition with just one bin.

Adding one different value to the saries works:

s = pd.Series([1] * 5 + [2])
pd.qcut(s, 3, duplicates='drop')

=>

0    (0.999, 2.0]
1    (0.999, 2.0]
2    (0.999, 2.0]
3    (0.999, 2.0]
4    (0.999, 2.0]
5    (0.999, 2.0]
dtype: category
Categories (1, interval[float64, right]): [(0.999, 2.0]]

Expected Behavior

A partition with just one bin would be a sensible answer when duplicates='drop' is required.

Installed Versions

INSTALLED VERSIONS

commit : 4bfe3d0
python : 3.9.12.final.0
python-bits : 64
OS : Darwin
OS-release : 20.6.0
Version : Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : en_US.UTF-8
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.4.2
numpy : 1.22.3
pytz : 2022.1
dateutil : 2.8.2
pip : 22.0.4
setuptools : 60.10.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.9.3
jinja2 : 3.1.2
IPython : 8.3.0
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
markupsafe : 2.1.1
matplotlib : 3.5.2
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.8.0
snappy : None
sqlalchemy : 1.4.36
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None

[simonjayhawkins]

Thanks @memeplex for the report.

does #16328 cover this?

A partition with just one bin would be a sensible answer when duplicates='drop' is required.

pd.qcut(s, 1, duplicates="drop") does work

maybe an error should be raised instead of producing the NaN result?

[memeplex]

I believe that when duplicates='drop' the most sensible behavior is to put everything into one bin, since that is what's done when just one bin is required:

s = pd.Series([1] * 5)
pd.qcut(s, 1, duplicates='drop')

=>

0    (0.999, 1.0]
1    (0.999, 1.0]
2    (0.999, 1.0]
3    (0.999, 1.0]
4    (0.999, 1.0]

That said, I must admit that I don't quite understand the behavior of duplicates='drop', there are some corner cases like this one:

s = pd.Series([1, 1, 1, 2, 2, 2])
pd.qcut(s, 2, duplicates='drop')

=>

0    (0.999, 1.5]
1    (0.999, 1.5]
2    (0.999, 1.5]
3      (1.5, 2.0]
4      (1.5, 2.0]
5      (1.5, 2.0]

but then

s = pd.Series([1, 1, 1, 2, 2])
pd.qcut(s, 2, duplicates='drop')

=>

0    (0.999, 2.0]
1    (0.999, 2.0]
2    (0.999, 2.0]
3    (0.999, 2.0]
4    (0.999, 2.0]

Is it joining both bins in the second example? That gives entirely different results for say [1] * 1000 + [2] * 999 vs [1] * 1000 + [2] * 1000, which seems odd to me. Does it makes sense to you?

[simonjayhawkins]

Does it makes sense to you?

I think where the result is not well defined it maybe should error. my reasoning here is that if labels=False, one would expect 3 bins labelled [0,1,2] in the result.

the most sensible behavior is to put everything into one bin

makes sense when the bins are intervals, but the result should probably be consistent with labels=False, where returning all 0's maybe does not make so much sense.

[memeplex]

Sorry, I'm not quite following you. For both [1] * 1000 + [2] * 999 vs [1] * 1000 + [2] * 1000, when requiring two bins I would have expected two bins, perhaps (0.999, 1.5] and (1.5, 2] as in the second example above. But perhaps I don't understand the underlying algorithm and it makes sense somehow that two similar vectors behave in such a different way. I tend to think about this as if you first ranked the vector.

[simonjayhawkins]

I tend to think about this as if you first ranked the vector.

yes this mentally makes sense and also means that labels=False should always be able to produce a result with the requested number of bins so long as the number of rows is greater than the number of requested bins, whether the values are unique or not.

I believe the implementation however is trying to calculate the boundaries and therefore fails (or produces unexpected results) with non-unique values.

from #16328 (comment)

I think this is a bug in the qcut logic with duplicates.

Specifically, qcut decides on the quantiles using linspace if it isn't specified. The linspace is np.linspace(0,1, num_quantiles+1). The bucket ranges are then constructed by taking consecutive pairs of the quantiles values.

The problem is if the min and first quantile values are duplicate, than we drop one and the first quantile is then treated as the min for the first bucket constructed.

I'm going to close this as a duplicate to avoid having similar discussion in multiple places.