Skip to content

BUG: pivot_table with nested elements and numpy 1.24 #50682

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Jan 17, 2023
Merged

BUG: pivot_table with nested elements and numpy 1.24 #50682

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
b43303f
Fix asarray_tuplesafe for numpy 1.24.1 deprecation
mroeschke Jan 11, 2023
750ac7d
BUG: pivot_table with nested elements and numpy 1.24
mroeschke Jan 11, 2023
1c25540
Merge remote-tracking branch 'upstream/main' into compat/np/pivot
mroeschke Jan 12, 2023
5c46695
For all numpy versions
mroeschke Jan 12, 2023
4b88e87
Undo unneeded variable
mroeschke Jan 12, 2023
c51966b
Merge remote-tracking branch 'upstream/main' into compat/np/pivot
mroeschke Jan 12, 2023
5c482eb
Merge remote-tracking branch 'upstream/main' into compat/np/pivot
mroeschke Jan 13, 2023
c56e38a
fix for arraymanager
mroeschke Jan 13, 2023
ec60be0
Merge remote-tracking branch 'upstream/main' into compat/np/pivot
mroeschke Jan 13, 2023
016cf09
Merge remote-tracking branch 'upstream/main' into compat/np/pivot
mroeschke Jan 13, 2023
01649f1
Merge remote-tracking branch 'upstream/main' into compat/np/pivot
mroeschke Jan 14, 2023
419f857
use try except
mroeschke Jan 15, 2023
dba09f4
Merge remote-tracking branch 'upstream/main' into compat/np/pivot
mroeschke Jan 16, 2023
b8f453b
typing
mroeschke Jan 16, 2023
b68e727
Update pandas/core/common.py
mroeschke Jan 17, 2023
cb084a6
line length
mroeschke Jan 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v1.5.3.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ Bug fixes
- Bug in :meth:`Series.quantile` emitting warning from NumPy when :class:`Series` has only ``NA`` values (:issue:`50681`)
- Bug when chaining several :meth:`.Styler.concat` calls, only the last styler was concatenated (:issue:`49207`)
- Fixed bug when instantiating a :class:`DataFrame` subclass inheriting from ``typing.Generic`` that triggered a ``UserWarning`` on python 3.11 (:issue:`49649`)
- Bug in :func:`pivot_table` with NumPy 1.24 or greater when the :class:`DataFrame` columns has nested elements (:issue:`50342`)
- Bug in :func:`pandas.testing.assert_series_equal` (and equivalent ``assert_`` functions) when having nested data and using numpy >= 1.25 (:issue:`50360`)
-

.. ---------------------------------------------------------------------------
.. _whatsnew_153.other:
Expand Down
5 changes: 4 additions & 1 deletion pandas/core/common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@
is_bool_dtype,
is_extension_array_dtype,
is_integer,
is_list_like,
)
from pandas.core.dtypes.generic import (
ABCExtensionArray,
Expand Down Expand Up @@ -232,7 +233,9 @@ def asarray_tuplesafe(values: Iterable, dtype: NpDtype | None = None) -> ArrayLi
elif isinstance(values, ABCIndex):
return values._values

if isinstance(values, list) and dtype in [np.object_, object]:
if isinstance(values, list) and (
dtype in [np.object_, object] or any(is_list_like(val) for val in values)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any idea about performance impact here?

Copy link
Member Author

@mroeschke mroeschke Jan 15, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah in the worst case where this can't short circuit there's a perf hit

In [1]: values = list(range(1000))

In [2]: %timeit pd.core.common.asarray_tuplesafe(values) # PR
279 µs ± 1.6 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In [2]: %timeit pd.core.common.asarray_tuplesafe(values) # main
41.8 µs ± 623 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a common use-case?

Would try-except be significantly faster?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. With the try except I get closer to main performance (~44)

):
return construct_1d_object_array_from_listlike(values)

result = np.asarray(values, dtype=dtype)
Expand Down
69 changes: 69 additions & 0 deletions pandas/tests/reshape/test_pivot.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2312,6 +2312,75 @@ def test_pivot_table_datetime_warning(self):
)
tm.assert_frame_equal(result, expected)

def test_pivot_table_with_mixed_nested_tuples(self, using_array_manager):
# GH 50342
df = DataFrame(
{
"A": ["foo", "foo", "foo", "foo", "foo", "bar", "bar", "bar", "bar"],
"B": ["one", "one", "one", "two", "two", "one", "one", "two", "two"],
"C": [
"small",
"large",
"large",
"small",
"small",
"large",
"small",
"small",
"large",
],
"D": [1, 2, 2, 3, 3, 4, 5, 6, 7],
"E": [2, 4, 5, 5, 6, 6, 8, 9, 9],
("col5",): [
"foo",
"foo",
"foo",
"foo",
"foo",
"bar",
"bar",
"bar",
"bar",
],
("col6", 6): [
"one",
"one",
"one",
"two",
"two",
"one",
"one",
"two",
"two",
],
(7, "seven"): [
"small",
"large",
"large",
"small",
"small",
"large",
"small",
"small",
"large",
],
}
)
result = pivot_table(
df, values="D", index=["A", "B"], columns=[(7, "seven")], aggfunc=np.sum
)
expected = DataFrame(
[[4.0, 5.0], [7.0, 6.0], [4.0, 1.0], [np.nan, 6.0]],
columns=Index(["large", "small"], name=(7, "seven")),
index=MultiIndex.from_arrays(
[["bar", "bar", "foo", "foo"], ["one", "two"] * 2], names=["A", "B"]
),
)
if using_array_manager:
# INFO(ArrayManager) column without NaNs can preserve int dtype
expected["small"] = expected["small"].astype("int64")
tm.assert_frame_equal(result, expected)


class TestPivot:
def test_pivot(self):
Expand Down