BUG: Even though dropna=True, Series.value_counts() with arguments normalize=True and bins=Union[int,IntervalIndex], it still counts pd.NA values.

[jeojr]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

import pandas as pd

data = [1]*25 + [2]*25 + [3]*25 + [pd.NA]*25

s = pd.Series(data, dtype=pd.Int64Dtype()) # 'Int64'

BINVALUE = 2 
#BINVALUE = 1                          # Tested
#BINVALUE = pd.interval_range(0,12,4)  # Tested

#assert pd.__version__ == '1.3.5'

# Yes, it has nans
assert s.hasnans

# normalize=False and bins - Works as it should.
assert s.dropna().value_counts(normalize=False, bins=BINVALUE, dropna=True).sum() == \
                s.value_counts(normalize=False, bins=BINVALUE, dropna=True).sum(), \
                'normalize=False and bins should not raise AssertionError'

# normalize=True and NO BINS  - Works as it should.
assert s.dropna().value_counts(normalize=True, dropna=True).sum() == \
                s.value_counts(normalize=True, dropna=True).sum(), \
                'normalize=True and NO BINS should not raise AssertionError'

# normalize=True and ANY bins - Not working as it should.
assert s.dropna().value_counts(normalize=True, bins=BINVALUE, dropna=True).sum() == \
                s.value_counts(normalize=True, bins=BINVALUE, dropna=True).sum(), \
                'normalize=True and ANY bins is not dropping pd.NA'

Issue Description

Note: this issue also happens with pandas.value_counts(). I actually noticed the problem when I tried to apply this to a DataFrame (with pd.Int64Dtype dtypes).

df.apply(pd.value_counts, bins=pd.interval_range(0,12,4), normalize=True)

As can be seen from the Reproducible Example, the issue only occurs if:

• normalize=True
• bins is not None

Expected Behavior

import pandas as pd

data = [1]*25 + [2]*25 + [3]*25 + [pd.NA]*25

s = pd.Series(data, dtype=pd.Int64Dtype()) # 'Int64'

BINS = [2, 1, pd.interval_range(0,12,4)]

# EXPECTED
for binvalue in BINS:
    _sum = s.value_counts(normalize=True, bins=binvalue).sum()  # dropna=True by default
    print(f"{_sum} == 1.0  ->  {_sum == 1.0} (Expected 1 but getting {_sum})")

# WORKAROUND (using dropna() before value_counts())
for binvalue in BINS:
    dropna_prefixed = s.dropna().value_counts(normalize=True, bins=binvalue)
    print(f"{dropna_prefixed.sum()} == 1.0  ->  {dropna_prefixed.sum() == 1.0} (Workaround OK)")

Expected Behaviour Output

0.75 == 1.0 -> False (Expected 1 getting 0.75)
0.75 == 1.0 -> False (Expected 1 getting 0.75)
0.75 == 1.0 -> False (Expected 1 getting 0.75)
1.0 == 1.0 -> True (Workaround OK)
1.0 == 1.0 -> True (Workaround OK)
1.0 == 1.0 -> True (Workaround OK)

Installed Versions

INSTALLED VERSIONS

commit : 66e3805
python : 3.9.9.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.156-1.fc25.qubes.x86_64
Version : #1 SMP Fri Oct 29 02:51:34 CEST 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.3.5
numpy : 1.21.4
pytz : 2021.3
dateutil : 2.8.2
pip : 21.3.1
setuptools : 58.5.3
Cython : None
pytest : 6.2.5
hypothesis : None
sphinx : 4.3.1
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.7.1
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.0.3
IPython : 7.30.1
pandas_datareader: None
bs4 : 4.10.0
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.5.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.7.3
sqlalchemy : 1.4.27
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[tarsur909]

take