Skip to content

PERF: RangeIndex cache is written when calling df.loc[array] #53387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
3 tasks done
GianlucaFicarelli opened this issue May 25, 2023 · 2 comments · Fixed by #53397
Closed
3 tasks done

PERF: RangeIndex cache is written when calling df.loc[array] #53387

GianlucaFicarelli opened this issue May 25, 2023 · 2 comments · Fixed by #53397
Assignees
Labels
Indexing Related to indexing on series/frames, not to indexes themselves Performance Memory or execution speed performance

Comments

@GianlucaFicarelli
Copy link
Contributor

GianlucaFicarelli commented May 25, 2023

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this issue exists on the latest version of pandas.

  • I have confirmed this issue exists on the main branch of pandas.

Reproducible Example

The cache used by RangeIndex is written when accessing the _data or _values properties.

This doesn't happen when calling df.loc[k] and k is a scalar.
However, this happens when calling df.loc[a] and a is an array.

This isn't usually a big issue, but if the RangeIndex is big enough, it's possible to see that the first access is slower than expected.
I think that in general the creation of the RangeIndex cache should be avoided unless needed.

Example:

import numpy as np
import pandas as pd

df = pd.DataFrame(index=pd.RangeIndex(400_000_000))

%time x = df.loc[200_000_000]  # fast
print("_data" in df.index._cache)  # False
%time x = df.loc[[200_000_000]]  # slow on the first execution
print("_data" in df.index._cache)  # True
%time x = df.loc[[200_000_000]]  # fast after the first execution

Result:

CPU times: user 44 µs, sys: 73 µs, total: 117 µs
Wall time: 122 µs
False
CPU times: user 221 ms, sys: 595 ms, total: 815 ms
Wall time: 815 ms
True
CPU times: user 356 µs, sys: 165 µs, total: 521 µs
Wall time: 530 µs

I can open a PR with a possible approach to calculate the result without loading the cache, that should have better performance on the first access, and similar performance after that.

Installed Versions

INSTALLED VERSIONS

commit : 37ea63d
python : 3.10.8.final.0
python-bits : 64
OS : Linux
OS-release : 3.10.0-1160.42.2.el7.x86_64
Version : #1 SMP Tue Aug 31 20:15:00 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : en_US.UTF-8
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.0.1
numpy : 1.24.3
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.7.2
pip : 23.1.2
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 8.13.2
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.7.1
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.1
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

Prior Performance

No response

@GianlucaFicarelli GianlucaFicarelli added Needs Triage Issue that has not been reviewed by a pandas team member Performance Memory or execution speed performance labels May 25, 2023
@jbrockmendel
Copy link
Member

I can open a PR with a possible approach to calculate the result without loading the cache, that should have better performance on the first access, and similar performance after that.

I'd be interested to see this.

@topper-123 topper-123 added the Indexing Related to indexing on series/frames, not to indexes themselves label May 26, 2023
@GianlucaFicarelli
Copy link
Contributor Author

take

@lithomas1 lithomas1 removed the Needs Triage Issue that has not been reviewed by a pandas team member label May 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Indexing Related to indexing on series/frames, not to indexes themselves Performance Memory or execution speed performance
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants