Skip to content

BUG: Groupby quantiles incorrect bins #33200 #33644

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
May 25, 2020
Merged

BUG: Groupby quantiles incorrect bins #33200 #33644

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
15a27ea
BUG: Maintain the order of the bins in group_quantile. Updated tests …
mabelvj Apr 18, 2020
8e6af0e
BUG: Fix quantile calculation #33200
mabelvj Apr 19, 2020
5832ba9
BUG: Fix quantile calculation. Only move -1 labels if there are any l…
mabelvj Apr 26, 2020
662c102
Merge branch 'master' into 33200-groupby-quantile
mabelvj Apr 26, 2020
e70e3ca
BUG: Explicit quantile to 0.5. Fix format #33200
mabelvj Apr 28, 2020
b0c309b
BUG: Fix whatsnew. #33200
mabelvj Apr 29, 2020
131a8c1
BUG: Fix merge master. #33200
mabelvj May 11, 2020
d18bdc4
Merge remote-tracking branch 'upstream/master' into 33200-groupby-qua…
simonjayhawkins May 15, 2020
e7c0eac
troubleshoot ci timeouts
simonjayhawkins May 15, 2020
07fa080
issue number and coments
simonjayhawkins May 15, 2020
24779c6
Merge remote-tracking branch 'upstream/master' into 33200-groupby-qua…
simonjayhawkins May 19, 2020
c6f41a4
add more test cases
simonjayhawkins May 19, 2020
6b28d91
Merge remote-tracking branch 'upstream/master' into 33200-groupby-qua…
simonjayhawkins May 20, 2020
f21e332
update test per comment
simonjayhawkins May 20, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.1.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -802,6 +802,7 @@ Groupby/resample/rolling
- Bug in :meth:`DataFrame.resample` where an ``AmbiguousTimeError`` would be raised when the resulting timezone aware :class:`DatetimeIndex` had a DST transition at midnight (:issue:`25758`)
- Bug in :meth:`DataFrame.groupby` where a ``ValueError`` would be raised when grouping by a categorical column with read-only categories and ``sort=False`` (:issue:`33410`)
- Bug in :meth:`GroupBy.first` and :meth:`GroupBy.last` where None is not preserved in object dtype (:issue:`32800`)
- Bug in :meth:`GroupBy.quantile` causes the quantiles to be shifted when the ``by`` axis contains ``NaN`` (:issue:`33200`)
- Bug in :meth:`Rolling.min` and :meth:`Rolling.max`: Growing memory usage after multiple calls when using a fixed window (:issue:`30726`)

Reshaping
Expand Down
4 changes: 4 additions & 0 deletions pandas/_libs/groupby.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -778,6 +778,10 @@ def group_quantile(ndarray[float64_t] out,
if not mask[i]:
non_na_counts[lab] += 1

if labels.any():
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this required for? Do we have a test case when this is False?

Copy link
Contributor Author

@mabelvj mabelvj Apr 26, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Normally, if labels are not provided or the dataframe is empty, an error message will rise when applying the quantile. However, there are cases where certain operations lead to empty labels, as when time resampling some types of empty dataframe:

Here is an example of the pipeline:
https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=34258&view=logs&j=bef1c175-2c1b-51ae-044a-2437c76fc339&t=770e7bb1-09f5-5ebf-b63b-578d2906aac9&l=127

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this accounted for in the test that you've created? If not, can you add a test for it?

Copy link
Member

@simonjayhawkins simonjayhawkins May 20, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add two cases that take the if labels.any(): is False path.

What is this required for?

otherwise labels.max() can raise ValueError: zero-size array to reduction operation maximum which has no identity

# Put '-1' (NaN) labels as the last group so it does not interfere
# with the calculations.
labels[labels == -1] = np.max(labels) + 1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it appears that modifying labels is causing a segfault in test_quantile_missing_group_values_no_segfaults on windows on the second call in the loop. I suspect this is the reason for the timeouts.

# Get an index of values sorted by labels and then values
order = (values, labels)
sort_arr = np.lexsort(order).astype(np.int64, copy=False)
Expand Down
21 changes: 15 additions & 6 deletions pandas/tests/groupby/test_quantile.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,14 +181,23 @@ def test_quantile_missing_group_values_no_segfaults():
grp.quantile()


def test_quantile_missing_group_values_correct_results():
# GH 28662
data = np.array([1.0, np.nan, 3.0, np.nan])
df = pd.DataFrame(dict(key=data, val=range(4)))
@pytest.mark.parametrize(
"key, val, expected_key, expected_val",
[
([1.0, np.nan, 3.0, np.nan], range(4), [1.0, 3.0], [0.0, 2.0]),
([1.0, np.nan, 2.0, 2.0], range(4), [1.0, 2.0], [0.0, 2.5]),
(["a", "b", "b", np.nan], range(4), ["a", "b"], [0, 1.5]),
],
)
def test_quantile_missing_group_values_correct_results(
key, val, expected_key, expected_val
):
# GH 28662, GH 33200
df = pd.DataFrame({"key": key, "val": val})

result = df.groupby("key").quantile()
result = df.groupby("key").quantile(0.5)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add another test that does not explicitly set a quantile (e.g. like the original)

expected = pd.DataFrame(
[1.0, 3.0], index=pd.Index([1.0, 3.0], name="key"), columns=["val"]
expected_val, index=pd.Index(expected_key, name="key"), columns=["val"]
)
tm.assert_frame_equal(result, expected)

Expand Down