Skip to content

min_count is ignored by sum/prod when resampling a PeriodIndex #19974

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
roedema opened this issue Mar 2, 2018 · 2 comments · Fixed by #31196
Closed

min_count is ignored by sum/prod when resampling a PeriodIndex #19974

roedema opened this issue Mar 2, 2018 · 2 comments · Fixed by #31196
Labels
good first issue Needs Tests Unit test(s) needed to prevent regressions
Milestone

Comments

@roedema
Copy link

roedema commented Mar 2, 2018

Code Sample

Setup:

import numpy as np
import pandas as pdindex = pd.date_range(start='2018', freq='M', periods=6)
data = np.ones(6)
data[3:6] = np.nan

datetime = pd.Series(data, index)
period = datetime.to_period()

Resampling and summing (min_count=1) the Series with a DatetimeIndex:

datetime
datetime.resample('Q').sum(min_count=1)
2018-01-31   1.0
2018-02-28   1.0
2018-03-31   1.0
2018-04-30   NaN
2018-05-31   NaN
2018-06-30   NaN
Freq: M, dtype: float64
2018-03-31    3.0
2018-06-30    NaN
Freq: Q-DEC, dtype: float64

Resampling and summing (min_count=1) the Series with a PeriodIndex:

period
period.resample('Q').sum(min_count=1)
2018-01    1.0
2018-02    1.0
2018-03    1.0
2018-04    NaN
2018-05    NaN
2018-06    NaN
Freq: M, dtype: float64
2018Q1    3.0
2018Q2    0.0
Freq: Q-DEC, dtype: float64

Problem description

sum() and prod() seem to ignore the min_count argument when used on a resampled series or dataframe with a PeriodIndex.

Expected Output

I would expect the same result whether using a DatetimeIndex or PeriodIndex. Specifically, in the example above, 2018Q2 should be NaN.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 61 Stepping 4, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.22.0
pytest: 3.2.1
pip: 9.0.1
setuptools: 36.5.0.post20170921
Cython: 0.26.1
numpy: 1.13.3
scipy: 0.19.1
pyarrow: None
xarray: None
IPython: 6.1.0
sphinx: 1.6.3
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: 2.1.0
openpyxl: 2.4.8
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.2
lxml: 4.1.0
bs4: 4.6.0
html5lib: 0.999999999
sqlalchemy: 1.1.13
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

@roedema roedema changed the title min_count is ignored by sum/prod when resampling min_count is ignored by sum/prod when resampling a PeriodIndex Mar 2, 2018
@gfyoung gfyoung added Bug Indexing Related to indexing on series/frames, not to indexes themselves Period Period data type labels Mar 8, 2018
@gfyoung
Copy link
Member

gfyoung commented Mar 8, 2018

Indeed, this does look like a violation of the spec. Investigation and patch is welcome!

@mroeschke
Copy link
Member

Looks to be fixed on master. Could use a test.

In [3]: period.resample('Q').sum(min_count=1)
   ...:
Out[3]:
2018Q1    3.0
2018Q2    NaN
Freq: Q-DEC, dtype: float64

In [4]: pd.__version__
Out[4]: '0.26.0.dev0+593.g9d45934af'

@mroeschke mroeschke added good first issue Needs Tests Unit test(s) needed to prevent regressions and removed Bug Indexing Related to indexing on series/frames, not to indexes themselves Period Period data type labels Oct 20, 2019
@jreback jreback added this to the 1.0 milestone Nov 22, 2019
@jreback jreback modified the milestones: 1.0, Contributions Welcome Jan 1, 2020
@simonjayhawkins simonjayhawkins modified the milestones: Contributions Welcome, 1.1 Jan 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Needs Tests Unit test(s) needed to prevent regressions
Projects
None yet
5 participants