Skip to content

BUG: value_counts not working correctly on ExtensionArrays #33674

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
May 1, 2020
Merged

BUG: value_counts not working correctly on ExtensionArrays #33674

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
11c26e2
BUG: value_counts not working correctly on ExtensionArrays
kotamatsuoka Apr 20, 2020
b025232
Change sum() to ndarray.sum()
kotamatsuoka Apr 20, 2020
7007205
Expand tests
kotamatsuoka Apr 21, 2020
af97126
Update test
kotamatsuoka Apr 22, 2020
cbabcad
Resolve warning
kotamatsuoka Apr 24, 2020
dac1af5
Pass test in Windows py36_np15
kotamatsuoka Apr 24, 2020
0be33fc
Change Int32 to Int64
kotamatsuoka Apr 25, 2020
660bba3
Merge branch 'master' into normalize-value-counts
kotamatsuoka Apr 25, 2020
d5eec76
Fix docs
kotamatsuoka Apr 25, 2020
db8ea2e
Remove product code update
kotamatsuoka Apr 25, 2020
243d3ec
Remove unnecessary changes
kotamatsuoka Apr 26, 2020
67ee980
split new test
kotamatsuoka Apr 27, 2020
405037f
Update normalize value_counts test
kotamatsuoka Apr 29, 2020
5d54a05
add coments
kotamatsuoka Apr 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pandas/core/arrays/string_.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -290,7 +290,7 @@ def _reduce(self, name, skipna=True, **kwargs):
def value_counts(self, dropna=False):
from pandas import value_counts

return value_counts(self._ndarray, dropna=dropna).astype("Int64")
return value_counts(self._ndarray, dropna=dropna)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you revert this back to what it was on master (will need to change the string tests back as well)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If revert, value_counts method return different dtype and need to change many tests. (comment)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dsaxton why/where were these changed to Int64 in the first place?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like it was added here #30824

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jreback this is converted to Int64 on purpose, in the idea that operations on nullable columns should, to the extent possible, also result in new nullable dtypes (similarly as doing a comparison operation with string dtype will give a nullable boolean dtype)


# Override parent because we have different return types.
@classmethod
Expand Down
4 changes: 2 additions & 2 deletions pandas/tests/arrays/string_/test_string.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -297,9 +297,9 @@ def test_arrow_roundtrip():
def test_value_counts_na():
arr = pd.array(["a", "b", "a", pd.NA], dtype="string")
result = arr.value_counts(dropna=False)
expected = pd.Series([2, 1, 1], index=["a", "b", pd.NA], dtype="Int64")
expected = pd.Series([2, 1, 1], index=["a", "b", pd.NA])
tm.assert_series_equal(result, expected)

result = arr.value_counts(dropna=True)
expected = pd.Series([2, 1], index=["a", "b"], dtype="Int64")
expected = pd.Series([2, 1], index=["a", "b"])
tm.assert_series_equal(result, expected)
11 changes: 10 additions & 1 deletion pandas/tests/extension/base/methods.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ class BaseMethodsTests(BaseExtensionTests):

@pytest.mark.parametrize("dropna", [True, False])
def test_value_counts(self, all_data, dropna):
all_data = all_data[:10]
all_data = all_data[:10].unique()
if dropna:
other = np.array(all_data[~all_data.isna()])
else:
Expand All @@ -28,6 +28,15 @@ def test_value_counts(self, all_data, dropna):

self.assert_series_equal(result, expected)

result = (
pd.Series(all_data, dtype=all_data.dtype)
.value_counts(dropna=dropna, normalize=True)
.sort_index()
)

expected = pd.Series([1 / len(other)] * len(other), index=result.index)
self.assert_series_equal(result, expected)

def test_count(self, data_missing):
df = pd.DataFrame({"A": data_missing})
result = df.count(axis="columns")
Expand Down
4 changes: 1 addition & 3 deletions pandas/tests/extension/test_string.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,9 +90,7 @@ def test_reduce_series_numeric(self, data, all_numeric_reductions, skipna):


class TestMethods(base.BaseMethodsTests):
@pytest.mark.skip(reason="returns nullable")
def test_value_counts(self, all_data, dropna):
return super().test_value_counts(all_data, dropna)
pass


class TestCasting(base.BaseCastingTests):
Expand Down