ENH: RangeIndex._shallow_copy can return RangeIndex #47557

mroeschke · 2022-07-01T00:17:47Z

closes Index is being materialized in pd.concat when axis=1 #46675 (Replace xxxx with the Github issue number)
closes PERF: Index.union is materializing even when it is not needed when sort=False #43885
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added an entry in the latest doc/source/whatsnew/v1.5.0.rst file if fixing a bug or adding a new feature.

pandas/core/indexes/range.py

jreback

asv's prove out the change in perf?

pandas/core/indexes/range.py

mroeschke · 2022-07-08T16:53:00Z

asv's prove out the change in perf?

The memory perf will just be evident in the result

In [1]: >>> idx1 = pd.RangeIndex(0, 100_000)
   ...: >>> idx2 = pd.RangeIndex(0, 50_000)

# main
In [2]: idx1.union(idx2, sort=False).memory_usage()
Out[2]: 800000

# PR
In [2]: idx1.union(idx2, sort=False).memory_usage()
Out[2]: 128

jbrockmendel · 2022-07-08T17:57:10Z

idx1.union

no objection to this PR, but ideally this case would be caught in _union to prevent ever materializing the ndarray

mroeschke · 2022-07-08T21:43:27Z

idx1.union

no objection to this PR, but ideally this case would be caught in _union to prevent ever materializing the ndarray

Added logic to do this in 63e7894, but I think the change in _shallow_copy is still needed because concat goes though there

jreback

and asv's still hold here (after latest changes)?

mroeschke · 2022-07-08T23:48:59Z

and asv's still hold here (after latest changes)?

% asv continuous -f 1.1 upstream/main HEAD -b index_object.Range
· Creating environments.........................
· Discovering benchmarks.
·· Uninstalling from conda-py3.8-Cython0.29.30-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pyarrow-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt.
·· Building 498b94f7 <enh/range_shallowcopy> for conda-py3.8-Cython0.29.30-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pyarrow-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt...........................................................................
·· Installing 498b94f7 <enh/range_shallowcopy> into conda-py3.8-Cython0.29.30-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pyarrow-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt..
· Running 20 total benchmarks (2 commits * 1 environments * 10 benchmarks)
[  0.00%] · For pandas commit 8e6ca289 <main> (round 1/2):
[  0.00%] ·· Building for conda-py3.8-Cython0.29.30-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pyarrow-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt.........................................................................
[  0.00%] ·· Benchmarking conda-py3.8-Cython0.29.30-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pyarrow-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[  2.50%] ··· Running (index_object.Range.time_get_loc_dec--)..........
[ 25.00%] · For pandas commit 498b94f7 <enh/range_shallowcopy> (round 1/2):
[ 25.00%] ·· Building for conda-py3.8-Cython0.29.30-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pyarrow-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt..
[ 25.00%] ·· Benchmarking conda-py3.8-Cython0.29.30-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pyarrow-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 27.50%] ··· Running (index_object.Range.time_get_loc_dec--)..........
[ 50.00%] · For pandas commit 498b94f7 <enh/range_shallowcopy> (round 2/2):
[ 50.00%] ·· Benchmarking conda-py3.8-Cython0.29.30-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pyarrow-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 52.50%] ··· index_object.Range.time_get_loc_dec                                                                                                      554±100ns
[ 55.00%] ··· index_object.Range.time_get_loc_inc                                                                                                       509±50ns
[ 57.50%] ··· index_object.Range.time_iter_dec                                                                                                          14.4±2ms
[ 60.00%] ··· index_object.Range.time_iter_inc                                                                                                          15.9±1ms
[ 62.50%] ··· index_object.Range.time_max                                                                                                            1.20±0.09μs
[ 65.00%] ··· index_object.Range.time_max_trivial                                                                                                     1.10±0.2μs
[ 67.50%] ··· index_object.Range.time_min                                                                                                             1.24±0.1μs
[ 70.00%] ··· index_object.Range.time_min_trivial                                                                                                     1.48±0.3μs
[ 72.50%] ··· index_object.Range.time_sort_values_asc                                                                                                 2.73±0.2μs
[ 75.00%] ··· index_object.Range.time_sort_values_des                                                                                                   4.48±2μs
[ 75.00%] · For pandas commit 8e6ca289 <main> (round 2/2):
[ 75.00%] ·· Building for conda-py3.8-Cython0.29.30-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pyarrow-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt...
[ 75.00%] ·· Benchmarking conda-py3.8-Cython0.29.30-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pyarrow-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 77.50%] ··· index_object.Range.time_get_loc_dec                                                                                                       550±60ns
[ 80.00%] ··· index_object.Range.time_get_loc_inc                                                                                                      506±100ns
[ 82.50%] ··· index_object.Range.time_iter_dec                                                                                                          12.8±1ms
[ 85.00%] ··· index_object.Range.time_iter_inc                                                                                                          13.9±3ms
[ 87.50%] ··· index_object.Range.time_max                                                                                                             1.34±0.2μs
[ 90.00%] ··· index_object.Range.time_max_trivial                                                                                                    1.13±0.06μs
[ 92.50%] ··· index_object.Range.time_min                                                                                                             1.19±0.3μs
[ 95.00%] ··· index_object.Range.time_min_trivial                                                                                                    1.09±0.08μs
[ 97.50%] ··· index_object.Range.time_sort_values_asc                                                                                                 3.03±0.8μs
[100.00%] ··· index_object.Range.time_sort_values_des                                                                                                 5.22±0.5μs

BENCHMARKS NOT SIGNIFICANTLY CHANGED.

mroeschke · 2022-07-09T00:12:04Z

Thanks!

ENH: RangeIndex._shallow_copy can return RangeIndex

a14be3a

jbrockmendel reviewed Jul 1, 2022

View reviewed changes

pandas/core/indexes/range.py Outdated Show resolved Hide resolved

jreback added the Index Related to the Index class or subclasses label Jul 3, 2022

mroeschke added 6 commits July 5, 2022 11:20

Merge remote-tracking branch 'upstream/main' into enh/range_shallowcopy

92f60b1

unique_deltas

c7dde8c

Merge remote-tracking branch 'upstream/main' into enh/range_shallowcopy

ba0b3b5

off by diff

76fbb72

Merge remote-tracking branch 'upstream/main' into enh/range_shallowcopy

8aeb814

Add whatsnew and address 0 diff

0c1d519

mroeschke added this to the 1.5 milestone Jul 6, 2022

Add additional unit test

7d5d216

jreback requested changes Jul 8, 2022

View reviewed changes

pandas/core/indexes/range.py Show resolved Hide resolved

Merge remote-tracking branch 'upstream/main' into enh/range_shallowcopy

2714076

Add comment

da4af1d

Still return rangeindex if sort=False

63e7894

jreback approved these changes Jul 8, 2022

View reviewed changes

jreback merged commit e915b0a into pandas-dev:main Jul 8, 2022

mroeschke deleted the enh/range_shallowcopy branch July 9, 2022 00:12

yehoshuadimarsky pushed a commit to yehoshuadimarsky/pandas that referenced this pull request Jul 13, 2022

ENH: RangeIndex._shallow_copy can return RangeIndex (pandas-dev#47557)

27f69dd

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: RangeIndex._shallow_copy can return RangeIndex #47557

ENH: RangeIndex._shallow_copy can return RangeIndex #47557

mroeschke commented Jul 1, 2022 •

edited

Loading

jreback left a comment

mroeschke commented Jul 8, 2022

jbrockmendel commented Jul 8, 2022

mroeschke commented Jul 8, 2022

jreback left a comment

mroeschke commented Jul 8, 2022

mroeschke commented Jul 9, 2022

ENH: RangeIndex._shallow_copy can return RangeIndex #47557

ENH: RangeIndex._shallow_copy can return RangeIndex #47557

Conversation

mroeschke commented Jul 1, 2022 • edited Loading

jreback left a comment

Choose a reason for hiding this comment

mroeschke commented Jul 8, 2022

jbrockmendel commented Jul 8, 2022

mroeschke commented Jul 8, 2022

jreback left a comment

Choose a reason for hiding this comment

mroeschke commented Jul 8, 2022

mroeschke commented Jul 9, 2022

mroeschke commented Jul 1, 2022 •

edited

Loading