Skip to content

PERF: concat(axis=1) with unaligned indexes #55084

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Sep 11, 2023

Conversation

lukemanley
Copy link
Member

       before           after         ratio
-      23.4±0.7ms         19.3±1ms     0.82  join_merge.ConcatIndexDtype.time_concat_series('string[python]', 'monotonic', 1, False)
-        54.7±6ms       42.6±0.3ms     0.78  join_merge.ConcatIndexDtype.time_concat_series('string[pyarrow]', 'has_na', 1, True)
-        19.4±2ms         14.4±1ms     0.75  join_merge.ConcatIndexDtype.time_concat_series('string[python]', 'has_na', 1, False)
-        40.2±3ms         29.8±1ms     0.74  join_merge.ConcatIndexDtype.time_concat_series('string[pyarrow]', 'monotonic', 1, False)
-        52.3±1ms       37.9±0.9ms     0.73  join_merge.ConcatIndexDtype.time_concat_series('string[pyarrow]', 'non_monotonic', 1, True)
-        37.6±3ms         24.5±1ms     0.65  join_merge.ConcatIndexDtype.time_concat_series('string[python]', 'has_na', 1, True)
-        6.81±1ms       4.35±0.4ms     0.64  join_merge.ConcatIndexDtype.time_concat_series('int64[pyarrow]', 'monotonic', 1, True)
-        7.56±1ms       4.81±0.3ms     0.64  join_merge.ConcatIndexDtype.time_concat_series('int64[pyarrow]', 'non_monotonic', 1, False)
-      39.1±0.8ms       24.6±0.2ms     0.63  join_merge.ConcatIndexDtype.time_concat_series('string[python]', 'non_monotonic', 1, True)
-       20.8±10ms       12.5±0.6ms     0.60  join_merge.ConcatIndexDtype.time_concat_series('string[python]', 'non_monotonic', 1, False)
-        9.35±2ms       5.56±0.6ms     0.59  join_merge.ConcatIndexDtype.time_concat_series('int64[pyarrow]', 'has_na', 1, False)
-        12.2±2ms       7.16±0.6ms     0.59  join_merge.ConcatIndexDtype.time_concat_series('int64[pyarrow]', 'has_na', 1, True)
-      7.09±0.8ms       4.00±0.9ms     0.56  join_merge.ConcatIndexDtype.time_concat_series('int64', 'non_monotonic', 1, False)
-        10.3±1ms       5.66±0.2ms     0.55  join_merge.ConcatIndexDtype.time_concat_series('int64[pyarrow]', 'non_monotonic', 1, True)
-        8.09±1ms       4.17±0.2ms     0.52  join_merge.ConcatIndexDtype.time_concat_series('int64[pyarrow]', 'monotonic', 1, False)
-        7.47±1ms       3.74±0.2ms     0.50  join_merge.ConcatIndexDtype.time_concat_series('Int64', 'has_na', 1, False)
-        9.98±1ms      4.99±0.08ms     0.50  join_merge.ConcatIndexDtype.time_concat_series('Int64', 'non_monotonic', 1, True)
-       59.3±20ms       29.0±0.4ms     0.49  join_merge.ConcatIndexDtype.time_concat_series('string[pyarrow]', 'non_monotonic', 1, False)
-        27.9±3ms       13.4±0.9ms     0.48  join_merge.ConcatIndexDtype.time_concat_series('string[python]', 'monotonic', 1, True)
-        7.96±1ms       3.72±0.2ms     0.47  join_merge.ConcatIndexDtype.time_concat_series('Int64', 'monotonic', 1, False)
-        8.59±2ms      3.93±0.08ms     0.46  join_merge.ConcatIndexDtype.time_concat_series('Int64', 'non_monotonic', 1, False)
-        7.88±1ms       3.60±0.2ms     0.46  join_merge.ConcatIndexDtype.time_concat_series('Int64', 'monotonic', 1, True)
-        7.19±1ms      2.97±0.06ms     0.41  join_merge.ConcatIndexDtype.time_concat_series('int64', 'monotonic', 1, True)
-        7.46±2ms       3.06±0.2ms     0.41  join_merge.ConcatIndexDtype.time_concat_series('int64', 'monotonic', 1, False)
-        11.3±3ms       4.54±0.3ms     0.40  join_merge.ConcatIndexDtype.time_concat_series('Int64', 'has_na', 1, True)
-        10.8±2ms       3.81±0.1ms     0.35  join_merge.ConcatIndexDtype.time_concat_series('int64', 'non_monotonic', 1, True)

@lukemanley lukemanley added Performance Memory or execution speed performance Reshaping Concat, Merge/Join, Stack/Unstack, Explode labels Sep 11, 2023
@lukemanley lukemanley added this to the 2.2 milestone Sep 11, 2023
@mroeschke mroeschke merged commit 8929630 into pandas-dev:main Sep 11, 2023
@mroeschke
Copy link
Member

Thanks @lukemanley

mroeschke pushed a commit to mroeschke/pandas that referenced this pull request Sep 11, 2023
* PERF: concat(axis=1) with unaligned indexes

* whatsnew
@lukemanley lukemanley deleted the perf-concat-axis-1 branch September 12, 2023 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Performance Memory or execution speed performance Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants