Skip to content

BUG/df.agg-with-df-with-missing-values-results-in-IndexError #58864

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 5, 2024
+32 −3
Merged

BUG/df.agg-with-df-with-missing-values-results-in-IndexError #58864

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8c6f34b
fix
abeltavares May 29, 2024
4f32bc5
improve and fix bug entry
abeltavares Jun 3, 2024
165e2f3
update
abeltavares Jun 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v3.0.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -454,6 +454,7 @@ Datetimelike
- Bug in :func:`date_range` where the last valid timestamp would sometimes not be produced (:issue:`56134`)
- Bug in :func:`date_range` where using a negative frequency value would not include all points between the start and end values (:issue:`56382`)
- Bug in :func:`tseries.api.guess_datetime_format` would fail to infer time format when "%Y" == "%H%M" (:issue:`57452`)
- Bug in :meth:`Dataframe.agg` with df with missing values resulting in IndexError (:issue:`58810`)
- Bug in :meth:`DatetimeIndex.is_year_start` and :meth:`DatetimeIndex.is_quarter_start` does not raise on Custom business days frequencies bigger then "1C" (:issue:`58664`)
- Bug in :meth:`DatetimeIndex.is_year_start` and :meth:`DatetimeIndex.is_quarter_start` returning ``False`` on double-digit frequencies (:issue:`58523`)
- Bug in setting scalar values with mismatched resolution into arrays with non-nanosecond ``datetime64``, ``timedelta64`` or :class:`DatetimeTZDtype` incorrectly truncating those scalars (:issue:`56410`)
Expand Down
9 changes: 6 additions & 3 deletions pandas/core/apply.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1824,12 +1824,15 @@ def relabel_result(
fun = [
com.get_callable_name(f) if not isinstance(f, str) else f for f in fun
]
col_idx_order = Index(s.index).get_indexer(fun)
s = s.iloc[col_idx_order]
col_idx_order = list(Index(s.index).get_indexer(fun))
col_idx_order = [i for i in col_idx_order if i != -1]
if col_idx_order:
s = s.iloc[col_idx_order]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
col_idx_order = list(Index(s.index).get_indexer(fun))
col_idx_order = [i for i in col_idx_order if i != -1]
if col_idx_order:
s = s.iloc[col_idx_order]
col_idx_order = Index(s.index).get_indexer(fun)
col_idx_order = col_idx_order[col_idx_order != -1]
s = s.iloc[col_idx_order]

Copy link
Contributor Author

@abeltavares abeltavares Jun 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That won't produce the expected behavior.
Take the "A" example in the docstring without the condition it will be:

foo  NaN
aab  NaN
bar  NaN
dat  NaN

Which is wrong.

This happens because:

  • col_idx_order is determined by Index(s.index).get_indexer(fun).
  • Since s only has one value with index ["mean"] and fun = ["max"], there is no match, so col_idx_order = [-1].
  • The code s = s.iloc[col_idx_order] results in an empty Series because -1 indicates no match, producing thr wrong behaviour.

Is this not supposed to be?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah OK then you can add back the if not col_idx_order.empty: condition

Copy link
Contributor Author

@abeltavares abeltavares Jun 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AttributeError: 'numpy.ndarray' object has no attribute 'empty'
The best way i find was to make it a list and check that way.

I guess we could use a boolean mask directly on the NumPy array returned by get_indexer applying only the valid indices.

  col_idx_order = Index(s.index).get_indexer(fun)
  valid_idx = col_idx_order != -1
  if valid_idx.any():
      s = s.iloc[col_idx_order[valid_idx]]

Let me know what you think.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure that solution works. Thanks.


# assign the new user-provided "named aggregation" as index names, and reindex
# it based on the whole user-provided names.
s.index = reordered_indexes[idx : idx + len(fun)]
if len(s) > 0:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if len(s) > 0:
if not s.empty:

s.index = reordered_indexes[idx : idx + len(fun)]
reordered_result_in_dict[col] = s.reindex(columns)
idx = idx + len(fun)
return reordered_result_in_dict
Expand Down
26 changes: 26 additions & 0 deletions pandas/tests/groupby/aggregate/test_aggregate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,32 @@ def test_agg_ser_multi_key(df):
tm.assert_series_equal(results, expected)


def test_agg_with_missing_values():
# GH#58810
missing_df = DataFrame(
{
"nan": [np.nan, np.nan, np.nan, np.nan],
"na": [pd.NA, pd.NA, pd.NA, pd.NA],
"nat": [pd.NaT, pd.NaT, pd.NaT, pd.NaT],
"none": [None, None, None, None],
"values": [1, 2, 3, 4],
}
)

result = missing_df.agg(x=("nan", "min"), y=("na", "min"), z=("values", "sum"))

expected = DataFrame(
{
"nan": [np.nan, np.nan, np.nan],
"na": [np.nan, np.nan, np.nan],
"values": [np.nan, np.nan, 10.0],
},
index=["x", "y", "z"],
)

tm.assert_frame_equal(result, expected)


def test_groupby_aggregation_mixed_dtype():
# GH 6212
expected = DataFrame(
Expand Down