Skip to content

BUG: fix nanmedian for CoW without bottleneck #55742

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Nov 17, 2023
Merged

BUG: fix nanmedian for CoW without bottleneck #55742

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7f0ff96
BUG: fix nanmedian for CoW without bottleneck
jorisvandenbossche Oct 28, 2023
336cb69
don't fill with 0 for nanmedian
jorisvandenbossche Oct 30, 2023
60eaf54
Merge remote-tracking branch 'upstream/main' into cow-bug-nanmedian
jorisvandenbossche Nov 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions pandas/core/nanops.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -757,6 +757,10 @@ def nanmedian(values, *, axis: AxisInt | None = None, skipna: bool = True, mask=
>>> nanops.nanmedian(s.values)
2.0
"""
# for floats without mask, the data already uses NaN as missing value
# indicator, and `mask` will be calculated from that below -> in those
# cases we never need to set NaN to the masked values
using_nan_sentinel = values.dtype.kind == "f" and mask is None

def get_median(x, _mask=None):
if _mask is None:
Expand All @@ -774,7 +778,7 @@ def get_median(x, _mask=None):
return res

dtype = values.dtype
values, mask = _get_values(values, skipna, mask=mask, fill_value=0)
values, mask = _get_values(values, skipna, mask=mask, fill_value=None)
if values.dtype.kind != "f":
if values.dtype == object:
# GH#34671 avoid casting strings to numeric
Expand All @@ -786,7 +790,9 @@ def get_median(x, _mask=None):
except ValueError as err:
# e.g. "could not convert string to float: 'a'"
raise TypeError(str(err)) from err
if mask is not None:
if not using_nan_sentinel and mask is not None:
if not values.flags.writeable:
values = values.copy()
Comment on lines +794 to +795
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that in theory, this copy might not be needed, and we could also ignore the read-only flag instead. The case that could still end up here is the masked Float64Dtype, where you do pass float64 data together with a mask. Although in that case, setting the values under the mask to NaN should I think always be "safe" at the moment (I don't think we already allow sharing data but not mask between FloatingArrays). But that's maybe something we can optimize later for Float64Dtype when we have a better idea of its memory model (and if sharing of data is definitely never possible)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now, looking a bit more, in practice we don't actually get here for FloatingArray at the moment, for two reasons:

  1. we don't yet mark the numpy arrays as read-only when accessing those (FloatingArray _data), so it's only for the numpy float64 dtype that we would get read-only data at the moment

  2. if there are missing values in the FloatArray, we actually already copy the values inside _get_values:

    pandas/pandas/core/nanops.py

    Lines 312 to 316 in 984d755

    if fill_value is not None:
    if mask.any():
    if datetimelike or _na_ok_dtype(dtype):
    values = values.copy()
    np.putmask(values, mask, fill_value)

Now, that seems double effort, and something we should fix, but that's for another issue (we are also setting with a fill_value of 0 here, which doesn't seem to make sense in the case of median, and then in nanmedian itself we either filter with the mask or set NaNs into it ..)

So I can also leave out this if not values.flags.writeable: check, because right now it will not be covered by any test / case in pandas, I think (unless you would use nanmedian directly)

values[mask] = np.nan

notempty = values.size
Expand Down