Skip to content

REGR: concat materializing index even if sort is not necessary #47508

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 30, 2022
Merged

REGR: concat materializing index even if sort is not necessary #47508

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ec3c663
REGR: concat materializing index even if sort is not necessary
phofl Jun 25, 2022
e12ada3
Add check for index
phofl Jun 28, 2022
885d19a
Add exact
phofl Jun 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v1.4.4.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ including other versions of pandas.

Fixed regressions
~~~~~~~~~~~~~~~~~
-
- Fixed regression in :func:`concat` materializing :class:`Index` during sorting even if :class:`Index` was already sorted (:issue:`47501`)
-

.. ---------------------------------------------------------------------------
Expand Down
2 changes: 1 addition & 1 deletion pandas/core/indexes/api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ def _get_combined_index(
index = union_indexes(indexes, sort=False)
index = ensure_index(index)

if sort:
if sort and not index.is_monotonic_increasing:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe better to just have safe_sort check this internally? (we do this in sort_values for example)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Index.is_monotonic_decreasing as well?

Also I think a separate PR will need to be made against 1.4.x. safe_sort_index below is available on main and not 1.4.x

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah sorry, should have mentioned this. I’ll clean this up after this is backported

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So to confirm, once this is merged into main and backported, you'll modify the backport branch to be compatable with the 1.4.x branch?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No this should be compatible with 1.4.x if I am not missing anything.

I will create a new pr after this is merged and will move the no op check to safe_sort_index

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay gotcha. So does is_monotonic_decreasing also need to be added too?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don’t think so, if you want to sort and it is decreasing, you would have to turn the Range index around basically. We could add some logic for this, but not for 1.4.x I think

index = safe_sort_index(index)
# GH 29879
if copy:
Expand Down
9 changes: 9 additions & 0 deletions pandas/tests/reshape/concat/test_index.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -387,3 +387,12 @@ def test_concat_with_levels_with_none_keys(self, levels):
msg = "levels supported only when keys is not None"
with pytest.raises(ValueError, match=msg):
concat([df1, df2], levels=levels)

def test_concat_range_index_result(self):
# GH#47501
df1 = DataFrame({"a": [1, 2]})
df2 = DataFrame({"b": [1, 2]})

result = concat([df1, df2], sort=True, axis=1)
expected = DataFrame({"a": [1, 2], "b": [1, 2]})
tm.assert_frame_equal(result, expected)