Skip to content

Move frequencies functions to cython #17746

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 63 commits into from
Jan 5, 2018
Merged
Show file tree
Hide file tree
Changes from 56 commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
b77d09d
Move tseries.frequencies funcs up to tslibs.frequencies
jbrockmendel Oct 2, 2017
d551246
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Oct 2, 2017
cb227dc
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Oct 10, 2017
6532f76
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Oct 14, 2017
b5198f3
more specific noqa, try to fix incorrect flake warning
jbrockmendel Oct 20, 2017
3bfbdf1
disable flake8 false-positives
jbrockmendel Oct 24, 2017
4b8cb92
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Oct 31, 2017
da8369f
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Nov 1, 2017
74258c0
flake8 fixup
jbrockmendel Nov 1, 2017
823db82
consolidate constants near top of file
jbrockmendel Nov 1, 2017
c1d8ec1
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Nov 3, 2017
551f047
remove unused variable
jbrockmendel Nov 3, 2017
026056f
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Nov 5, 2017
925aca9
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Nov 9, 2017
92b1e9e
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Nov 10, 2017
ae06978
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Nov 10, 2017
2a2155e
update imports per reviewer request
jbrockmendel Nov 10, 2017
4192c0d
trim namespace using libfreqs
jbrockmendel Nov 10, 2017
21e13dc
fix import
jbrockmendel Nov 10, 2017
43c971b
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Nov 10, 2017
67dbfe2
update imports
jbrockmendel Nov 10, 2017
0292fbe
fix import
jbrockmendel Nov 11, 2017
22f46ac
remove unused import
jbrockmendel Nov 11, 2017
893fef9
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Nov 11, 2017
bcc5ca1
fixup missing import
jbrockmendel Nov 11, 2017
147bdc7
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Nov 11, 2017
b54fe29
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Nov 13, 2017
7bcdbe0
fixup imports
jbrockmendel Nov 13, 2017
38b520c
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Nov 13, 2017
3430925
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Nov 26, 2017
0029303
Deprivatize, add typing per reviewer request
jbrockmendel Nov 26, 2017
957345b
check set inclusion instead of list inclusion
jbrockmendel Nov 26, 2017
49f0f72
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Nov 29, 2017
4c200b2
fix broken import
jbrockmendel Nov 29, 2017
aa0f988
dummy commit to force CI
jbrockmendel Nov 30, 2017
1d12f09
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Dec 3, 2017
6b57c23
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Dec 6, 2017
9d0e70d
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Dec 8, 2017
664d51e
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Dec 8, 2017
e565a3a
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Dec 11, 2017
2138285
centralize+deprivatize day/month constants in ccalendar
jbrockmendel Dec 11, 2017
7d3cd5c
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Dec 11, 2017
cb73bd6
fixup typo
jbrockmendel Dec 11, 2017
db3717b
add typing
jbrockmendel Dec 11, 2017
cbefe3e
remove unused imports
jbrockmendel Dec 11, 2017
d1c67bc
fixup typo broken import
jbrockmendel Dec 11, 2017
f60fe4d
de-privatize get_rule_month
jbrockmendel Dec 11, 2017
f250550
remove unused import
jbrockmendel Dec 11, 2017
ae9f628
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Dec 11, 2017
db44aa5
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Dec 12, 2017
02e3217
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Dec 12, 2017
23a3588
add types and docstrings
jbrockmendel Dec 20, 2017
3769095
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Dec 20, 2017
a7f063c
typo fixup bool-->bint
jbrockmendel Dec 20, 2017
3ee6e92
fix typing
jbrockmendel Dec 21, 2017
64ef864
whitespace fixup
jbrockmendel Dec 21, 2017
56d7949
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Dec 24, 2017
6d62e04
implement test_libfrequencies per request
jbrockmendel Dec 24, 2017
1d71924
extend docstring per request
jbrockmendel Dec 24, 2017
730b21b
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Dec 29, 2017
010a7a5
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Jan 4, 2018
a5d408e
Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…
jbrockmendel Jan 4, 2018
8129b1c
fix docstrings, rename _assert_depr
jbrockmendel Jan 4, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions pandas/_libs/tslibs/frequencies.pxd
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
# -*- coding: utf-8 -*-
# cython: profile=False

cpdef object get_rule_month(object source, object default=*)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are these used anywhere but in period.pyx (for cython usage)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a duplicate of it in tslibs.parsing that should be replaced with a cimport.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(things will be easier on my end if we can clean those up in a follow-up; im guessing you'll also suggest de-privatizing _get_rule_month)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's fine, pls add to the list.

cpdef get_freq_code(freqstr)
cpdef object get_freq(object freq)
cpdef str get_base_alias(freqstr)
cpdef int get_to_timestamp_base(int base)
cpdef str get_freq_str(base, mult=*)
302 changes: 301 additions & 1 deletion pandas/_libs/tslibs/frequencies.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,12 @@ cimport cython

import numpy as np
cimport numpy as np
from numpy cimport int64_t
np.import_array()

from util cimport is_integer_object
from util cimport is_integer_object, is_string_object

from ccalendar import MONTH_NUMBERS

# ----------------------------------------------------------------------
# Constants
Expand All @@ -23,6 +26,22 @@ _INVALID_FREQ_ERROR = "Invalid frequency: {0}"
# ---------------------------------------------------------------------
# Period codes


class FreqGroup(object):
FR_ANN = 1000
FR_QTR = 2000
FR_MTH = 3000
FR_WK = 4000
FR_BUS = 5000
FR_DAY = 6000
FR_HR = 7000
FR_MIN = 8000
FR_SEC = 9000
FR_MS = 10000
FR_US = 11000
FR_NS = 12000


# period frequency constants corresponding to scikits timeseries
# originals
_period_code_map = {
Expand Down Expand Up @@ -203,3 +222,284 @@ cpdef _period_str_to_code(freqstr):
return _period_code_map[freqstr]
except KeyError:
raise ValueError(_INVALID_FREQ_ERROR.format(freqstr))


cpdef str get_freq_str(base, mult=1):
"""
Return the summary string associated with this offset code, possibly
adjusted by a multiplier.

Parameters
----------
base : int (member of FreqGroup)

Returns
-------
freq_str : str

Examples
--------
>>> get_freq_str(1000)
'A-DEC'

>>> get_freq_str(2000, 2)
'2Q-DEC'

>>> get_freq_str("foo")
"""
code = _reverse_period_code_map.get(base)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a doc-string

if mult == 1:
return code
return str(mult) + code


cpdef str get_base_alias(freqstr):
"""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add Parameters, Returns

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return str? (or do we just use object)

Returns the base frequency alias, e.g., '5D' -> 'D'
"""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add parameters/returns.

return _base_and_stride(freqstr)[0]


cpdef int get_to_timestamp_base(int base):
"""
Return frequency code group used for base of to_timestamp against
frequency code.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add Parameters, Returns

Parameters
----------
base : int (member of FreqGroup)

Returns
-------
base : int

Example
-------
# Return day freq code against longer freq than day
>>> get_to_timestamp_base(get_freq_code('D')[0])
6000
>>> get_to_timestamp_base(get_freq_code('W')[0])
6000
>>> get_to_timestamp_base(get_freq_code('M')[0])
6000

# Return second freq code against hour between second
>>> get_to_timestamp_base(get_freq_code('H')[0])
9000
>>> get_to_timestamp_base(get_freq_code('S')[0])
9000
"""
if base < FreqGroup.FR_BUS:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

return FreqGroup.FR_DAY
if FreqGroup.FR_HR <= base <= FreqGroup.FR_SEC:
return FreqGroup.FR_SEC
return base
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a considtion for base < FR_NS. (and make these if/else).

the final entry should be a raise (as base is out of range).



cpdef object get_freq(object freq):
"""
Return frequency code of given frequency str.
If input is not string, return input as it is.

Example
-------
>>> get_freq('A')
1000

>>> get_freq('3A')
1000
"""
if is_string_object(freq):
base, mult = get_freq_code(freq)
freq = base
return freq


# ----------------------------------------------------------------------
# Frequency comparison

cpdef bint is_subperiod(source, target):
"""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prob can be cpdef?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

type

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cpdef bool

Returns True if downsampling is possible between source and target
frequencies

Parameters
----------
source : string or DateOffset
Frequency converting from
target : string or DateOffset
Frequency converting to

Returns
-------
is_subperiod : boolean
"""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

later on can you come back and add examples to functions w/o


if target is None or source is None:
return False
source = _maybe_coerce_freq(source)
target = _maybe_coerce_freq(target)

if _is_annual(target):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

want to de-privatize these (again can add to list)

if _is_quarterly(source):
return _quarter_months_conform(get_rule_month(source),
get_rule_month(target))
return source in {'D', 'C', 'B', 'M', 'H', 'T', 'S', 'L', 'U', 'N'}
elif _is_quarterly(target):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this could be encoded in a dict (the source in .... part), simpler and less error prone

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are used outside of _libs and aren't too strict about typing. Do we want passing something non-hashable to raise?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wasn't talking about the _is_quarterly functions themselves, rather then internal impl.

return source in {'D', 'C', 'B', 'M', 'H', 'T', 'S', 'L', 'U', 'N'}
elif _is_monthly(target):
return source in {'D', 'C', 'B', 'H', 'T', 'S', 'L', 'U', 'N'}
elif _is_weekly(target):
return source in {target, 'D', 'C', 'B', 'H', 'T', 'S', 'L', 'U', 'N'}
elif target == 'B':
return source in {'B', 'H', 'T', 'S', 'L', 'U', 'N'}
elif target == 'C':
return source in {'C', 'H', 'T', 'S', 'L', 'U', 'N'}
elif target == 'D':
return source in {'D', 'H', 'T', 'S', 'L', 'U', 'N'}
elif target == 'H':
return source in {'H', 'T', 'S', 'L', 'U', 'N'}
elif target == 'T':
return source in {'T', 'S', 'L', 'U', 'N'}
elif target == 'S':
return source in {'S', 'L', 'U', 'N'}
elif target == 'L':
return source in {'L', 'U', 'N'}
elif target == 'U':
return source in {'U', 'N'}
elif target == 'N':
return source in {'N'}


cpdef bint is_superperiod(source, target):
"""
Returns True if upsampling is possible between source and target
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

frequencies

Parameters
----------
source : string
Frequency converting from
target : string
Frequency converting to

Returns
-------
is_superperiod : boolean
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

"""
if target is None or source is None:
return False
source = _maybe_coerce_freq(source)
target = _maybe_coerce_freq(target)

if _is_annual(source):
if _is_annual(target):
return get_rule_month(source) == get_rule_month(target)

if _is_quarterly(target):
smonth = get_rule_month(source)
tmonth = get_rule_month(target)
return _quarter_months_conform(smonth, tmonth)
return target in {'D', 'C', 'B', 'M', 'H', 'T', 'S', 'L', 'U', 'N'}
elif _is_quarterly(source):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same (maybe even same dict)

return target in {'D', 'C', 'B', 'M', 'H', 'T', 'S', 'L', 'U', 'N'}
elif _is_monthly(source):
return target in {'D', 'C', 'B', 'H', 'T', 'S', 'L', 'U', 'N'}
elif _is_weekly(source):
return target in {source, 'D', 'C', 'B', 'H', 'T', 'S', 'L', 'U', 'N'}
elif source == 'B':
return target in {'D', 'C', 'B', 'H', 'T', 'S', 'L', 'U', 'N'}
elif source == 'C':
return target in {'D', 'C', 'B', 'H', 'T', 'S', 'L', 'U', 'N'}
elif source == 'D':
return target in {'D', 'C', 'B', 'H', 'T', 'S', 'L', 'U', 'N'}
elif source == 'H':
return target in {'H', 'T', 'S', 'L', 'U', 'N'}
elif source == 'T':
return target in {'T', 'S', 'L', 'U', 'N'}
elif source == 'S':
return target in {'S', 'L', 'U', 'N'}
elif source == 'L':
return target in {'L', 'U', 'N'}
elif source == 'U':
return target in {'U', 'N'}
elif source == 'N':
return target in {'N'}


cdef str _maybe_coerce_freq(code):
""" we might need to coerce a code to a rule_code
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cpdef? type

and uppercase it

Parameters
----------
source : string or DateOffset
Frequency converting from

Returns
-------
code : string
"""
assert code is not None
if getattr(code, '_typ', None) == 'dateoffset':
# i.e. isinstance(code, ABCDateOffset):
code = code.rule_code
return code.upper()


cdef bint _quarter_months_conform(str source, str target):
snum = MONTH_NUMBERS[source]
tnum = MONTH_NUMBERS[target]
return snum % 3 == tnum % 3


cdef bint _is_annual(str rule):
rule = rule.upper()
return rule == 'A' or rule.startswith('A-')


cdef bint _is_quarterly(str rule):
rule = rule.upper()
return rule == 'Q' or rule.startswith('Q-') or rule.startswith('BQ')


cdef bint _is_monthly(str rule):
rule = rule.upper()
return rule == 'M' or rule == 'BM'


cdef bint _is_weekly(str rule):
rule = rule.upper()
return rule == 'W' or rule.startswith('W-')


# ----------------------------------------------------------------------
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extra line?


cpdef object get_rule_month(object source, object default='DEC'):
"""
Return starting month of given freq, default is December.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doc-string

Parameters
----------
source : object
default : object (default "DEC")

Returns
-------
rule_month: object (usually string)

Example
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-> Examples

-------
>>> get_rule_month('D')
'DEC'

>>> get_rule_month('A-JAN')
'JAN'
"""
if hasattr(source, 'freqstr'):
source = source.freqstr
source = source.upper()
if '-' not in source:
return default
else:
return source.split('-')[1]
16 changes: 9 additions & 7 deletions pandas/_libs/tslibs/period.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,16 @@ cimport util
from util cimport is_period_object, is_string_object, INT32_MIN

from pandas._libs.missing cimport is_null_datetimelike

from timestamps import Timestamp
from timezones cimport is_utc, is_tzlocal, get_utcoffset, get_dst_info
from timedeltas cimport delta_to_nanoseconds

from ccalendar import MONTH_NUMBERS
from parsing import (parse_time_string, NAT_SENTINEL,
_get_rule_month)
from frequencies cimport get_freq_code
from frequencies cimport (get_freq_code, get_base_alias,
get_to_timestamp_base, get_freq_str,
get_rule_month)
from parsing import parse_time_string, NAT_SENTINEL
from resolution import resolution, Resolution
from nattype import nat_strings, NaT, iNaT
from nattype cimport _nat_scalar_rules, NPY_NAT
Expand Down Expand Up @@ -572,7 +574,7 @@ cdef class _Period(object):

if isinstance(freq, (int, tuple)):
code, stride = get_freq_code(freq)
freq = frequencies._get_freq_str(code, stride)
freq = get_freq_str(code, stride)

freq = frequencies.to_offset(freq)

Expand Down Expand Up @@ -630,7 +632,7 @@ cdef class _Period(object):
raise IncompatibleFrequency(msg.format(self.freqstr))
elif isinstance(other, offsets.DateOffset):
freqstr = other.rule_code
base = frequencies.get_base_alias(freqstr)
base = get_base_alias(freqstr)
if base == self.freq.rule_code:
ordinal = self.ordinal + other.n
return Period(ordinal=ordinal, freq=self.freq)
Expand Down Expand Up @@ -756,7 +758,7 @@ cdef class _Period(object):

if freq is None:
base, mult = get_freq_code(self.freq)
freq = frequencies.get_to_timestamp_base(base)
freq = get_to_timestamp_base(base)

base, mult = get_freq_code(freq)
val = self.asfreq(freq, how)
Expand Down Expand Up @@ -1149,7 +1151,7 @@ def _quarter_to_myear(year, quarter, freq):
if quarter <= 0 or quarter > 4:
raise ValueError('Quarter must be 1 <= q <= 4')

mnum = MONTH_NUMBERS[_get_rule_month(freq)] + 1
mnum = MONTH_NUMBERS[get_rule_month(freq)] + 1
month = (mnum + (quarter - 1) * 3) % 12 + 1
if month > mnum:
year -= 1
Expand Down
2 changes: 1 addition & 1 deletion pandas/_libs/tslibs/resolution.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ from timezones cimport (is_utc, is_tzlocal,
maybe_get_tz, get_dst_info, get_utcoffset)
from fields import build_field_sarray
from conversion import tz_convert
from ccalendar import DAYS, MONTH_ALIASES, int_to_weekday
from ccalendar import MONTH_ALIASES, int_to_weekday

from pandas._libs.properties import cache_readonly
from pandas._libs.tslib import Timestamp
Expand Down
Loading