Skip to content

BUG: Fix qcut for nullable integers #31440

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
Feb 2, 2020
Merged

BUG: Fix qcut for nullable integers #31440

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
a0d2d95
Add test
Jan 30, 2020
fdfb2e9
Cast BooleanArray to bool
Jan 30, 2020
a35c7a6
Update 1.1 whatsnew
Jan 30, 2020
e2d1a46
Update test
Jan 30, 2020
bee26b1
Use place holder for NA values
Jan 30, 2020
516dbe3
Issue number
Jan 30, 2020
3f09385
Cast to numpy
Jan 31, 2020
dde3b61
Merge branch 'master' into qcut-null-int
Jan 31, 2020
5579381
Move note
Jan 31, 2020
93ef631
Fixup
Jan 31, 2020
f0990b1
Don't check for BooleanArray
Jan 31, 2020
99e6080
Make test data deterministic
Jan 31, 2020
5694ab6
Update test scripts
Feb 1, 2020
e26a13b
Merge branch 'master' into qcut-null-int
Feb 1, 2020
1bf4cee
Move test_cut for nullable int
Feb 1, 2020
4eb9313
Just import pandas
Feb 1, 2020
6767ea4
Cast to float instead of object in test
Feb 2, 2020
fc0a0ba
Move conversion into _coerce_to_type
Feb 2, 2020
465bb0e
Merge branch 'master' into qcut-null-int
Feb 2, 2020
b67abad
Move comments
Feb 2, 2020
e5c2598
Remove empty file
Feb 2, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions doc/source/whatsnew/v1.1.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,9 +200,7 @@ Sparse
ExtensionArray
^^^^^^^^^^^^^^

-
-

- Bug where :meth:`qcut` would raise when passed a nullable integer. (:issue:`31389`)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this can go in the 1.0.1 whatsnew now that it's been created.


Other
^^^^^
Expand Down
7 changes: 5 additions & 2 deletions pandas/core/reshape/tile.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -391,10 +391,13 @@ def _bins_to_cuts(
bins = unique_bins

side = "left" if right else "right"
ids = ensure_int64(bins.searchsorted(x, side=side))
ids = ensure_int64(bins.searchsorted(np.where(~isna(x), x, -1), side=side))

if include_lowest:
ids[x == bins[0]] = 1
min_mask = x == bins[0]
if is_extension_array_dtype(min_mask):
min_mask = min_mask.fillna(False).astype(bool)
ids[min_mask] = 1

na_mask = isna(x) | (ids == len(bins)) | (ids == 0)
has_nas = na_mask.any()
Expand Down
11 changes: 11 additions & 0 deletions pandas/tests/arrays/test_integer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1074,6 +1074,17 @@ def test_cut(bins, right, include_lowest):
tm.assert_categorical_equal(result, expected)


@pytest.mark.parametrize("q", [2, 5, 10])
def test_qcut_nullable_integer(q, any_nullable_int_dtype):
arr = pd.array(np.random.randint(1, 100, 100), dtype=any_nullable_int_dtype)
arr[::2] = pd.NA

result = pd.qcut(arr, q)
expected = pd.qcut(arr.astype(float), q)

tm.assert_categorical_equal(result, expected)


# TODO(jreback) - these need testing / are broken

# shift
Expand Down