BUG: Incorrect IntervalIndex.is_overlapping

[2-5]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

a = [ 10, 15, 20, 25, 30, 35, 40, 45, 45, 50, 55, 60, 65, 70, 75, 80, 85 ]
b = [ 15, 20, 25, 30, 35, 40, 45, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90 ]
#                             ^^^^^^^^^^ (overlapping region)

idx_1 = pd.IntervalIndex.from_arrays(a, b)
print(idx_1.is_overlapping)
# prints True

idx_2 = pd.IntervalIndex.from_arrays(a[1:], b[1:])
print(idx_2.is_overlapping)
# prints False (INCORRECT)

idx_3 = pd.IntervalIndex.from_arrays(a[:-1], b[:-1])
print(idx_3.is_overlapping)
# prints False (INCORRECT)

Issue Description

IntervalIndex.is_overlapping is returning incorrect values.

Notice how for the full a/b arrays it returns that there are overlapping intervals (they are in the middle), but if you remove the first/last pair ([1:], [-1:]), it returns incorrectly that there are no overlapping intervals.

Expected Behavior

Correct value

Installed Versions

commit : 91111fd python : 3.10.8.final.0 python-bits : 64 OS : Windows OS-release : 10

pandas : 1.5.1
numpy : 1.23.4
pytz : 2022.5
dateutil : 2.8.2
setuptools : 65.5.0
pip : 22.3
Cython : 0.29.32
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.1
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.5.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli : 1.0.9
fastparquet : 0.8.3
fsspec : 2022.10.0
gcsfs : None
matplotlib : 3.6.1
numba : 0.56.3
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 10.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.9.3
snappy :
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : 0.19.0
tzdata : 2022.5

[Hikipeko]

take

[Hikipeko]

Hi, @2-5 thanks for the bug report.

I guess that the overlapping region doesn't exist since the from_arrays function return intervals that are open on the right and closed on the left. So the interval (45, 45] is an empty set and won't cause a problem.

However, if the above is true, idx_1.is_overlapping should return False instead of True. This should still be a bug.

import pandas as pd

a = [ 10, 15, 20, 25, 30, 35, 40, 45, 45, 50, 55, 60, 65, 70, 75, 80, 85 ]
b = [ 15, 20, 25, 30, 35, 40, 45, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90 ]
#                             ^^^^^^^^^^ (not overlapping)

idx_1 = pd.IntervalIndex.from_arrays(a, b)
print(idx_1.is_overlapping)
# prints True (INCORRECT)

idx_2 = pd.IntervalIndex.from_arrays(a[1:], b[1:])
print(idx_2.is_overlapping)
# prints False

idx_3 = pd.IntervalIndex.from_arrays(a[:-1], b[:-1])
print(idx_3.is_overlapping)
# prints False 

[2-5]

I wasn't sure what was the correct behavior. I only encountered the issue when having consecutive empty intervals - (45, 45], (45, 45], so I assumed that IntervalIndex doesn't allow duplication at all. If there is a single empty interval, the bug doesn't seem to occur.

I encountered the problem by passing a similar pd.IntervalIndex to pd.cut(..., bins=idx_1), which then raises an exception that overlapping intervals are not allowed.

[SlowMo24]

I had a similar issue where is_overlapping returned False but I could not drop a line from the DataFrame because it refused to drop with an overlapping index. The result of is_overlapping was random in that case: an IntervalIndex would return False but if recreated (with the same data except) with one element removed, it would return True.

I was able to create an small issue worthy example DataFrame with version 1.5.3 (current) but that example then no longer worked with the current main (although my issue persists there), which is required when filing a bug.

So my workaround is to pad these zero-length intervals by 1 millisecond (timedelta + timedelta(microseconds=1)) so the get >0 length (because I didn't want to delete them).

[jbrockmendel]

I was able to create an small issue worthy example DataFrame with version 1.5.3 (current) but that example then no longer worked with the current main (although my issue persists there), which is required when filing a bug.

When you say "no longer working" do you mean the bug is fixed on main? If not, go ahead and open an issue anyway.

[SlowMo24]

Thanks for reaching out. I looked deeper into it and this bug is in fact fixed in the main branch. Yet, another one came up and I have opened #52245 .