Skip to content

REF: avoid getattr pattern for diff_2d; use fused types #29120

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Oct 22, 2019
Merged

REF: avoid getattr pattern for diff_2d; use fused types #29120

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1dbb8ec
REF: Implement diff_2d using fused_types
jbrockmendel Oct 20, 2019
be0c066
Merge branch 'master' of https://github.com/pandas-dev/pandas into cy…
jbrockmendel Oct 20, 2019
2a6e34d
black fixup
jbrockmendel Oct 20, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 23 additions & 22 deletions pandas/_libs/algos_common_helper.pxi.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,32 +4,34 @@ Template for each `dtype` helper function using 1-d template
WARNING: DO NOT edit .pxi FILE directly, .pxi is generated from .pxi.in
"""

{{py:

# name, c_type, dest_type
dtypes = [('float64', 'float64_t', 'float64_t'),
('float32', 'float32_t', 'float32_t'),
('int8', 'int8_t', 'float32_t'),
('int16', 'int16_t', 'float32_t'),
('int32', 'int32_t', 'float64_t'),
('int64', 'int64_t', 'float64_t')]

def get_dispatch(dtypes):

for name, c_type, dest_type, in dtypes:
yield name, c_type, dest_type
ctypedef fused diff_t:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn’t this numeric? can we consolidate these later

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i hope so

float64_t
float32_t
int8_t
int16_t
int32_t
int64_t

}}

{{for name, c_type, dest_type
in get_dispatch(dtypes)}}
ctypedef fused out_t:
float32_t
float64_t


@cython.boundscheck(False)
@cython.wraparound(False)
def diff_2d_{{name}}(ndarray[{{c_type}}, ndim=2] arr,
ndarray[{{dest_type}}, ndim=2] out,
Py_ssize_t periods, int axis):
def diff_2d(ndarray[diff_t, ndim=2] arr,
ndarray[out_t, ndim=2] out,
Py_ssize_t periods, int axis):

# Disable for unsupported dtype combinations,
# see https://github.com/cython/cython/issues/2646
if out_t is float32_t:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assume this a later followup

if not (diff_t is float32_t or diff_t is int8_t or diff_t is int16_t):
raise NotImplementedError
else:
if (diff_t is float32_t or diff_t is int8_t or diff_t is int16_t):
raise NotImplementedError
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@WillAyd you may have noticed a bunch of build warnings about unreachable code have started showing up. I'm pretty sure this is the reason. Any thoughts on more graceful ways to get the combination of dtypes we're interested in? (see the GH link on L27)

The same dtype-combination thing is the only reason why algos_take_helper isn't converted to fused types; it'd be really nice to clear that up

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't know off the top of my head - maybe an Enum using the type as a member?


cdef:
Py_ssize_t i, j, sx, sy

Expand Down Expand Up @@ -69,7 +71,6 @@ def diff_2d_{{name}}(ndarray[{{c_type}}, ndim=2] arr,
for j in range(start, stop):
out[i, j] = arr[i, j] - arr[i, j - periods]

{{endfor}}

# ----------------------------------------------------------------------
# ensure_dtype
Expand Down
14 changes: 7 additions & 7 deletions pandas/core/algorithms.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1860,12 +1860,12 @@ def searchsorted(arr, value, side="left", sorter=None):
# ---- #

_diff_special = {
"float64": algos.diff_2d_float64,
"float32": algos.diff_2d_float32,
"int64": algos.diff_2d_int64,
"int32": algos.diff_2d_int32,
"int16": algos.diff_2d_int16,
"int8": algos.diff_2d_int8,
"float64",
"float32",
"int64",
"int32",
"int16",
"int8",
}


Expand Down Expand Up @@ -1914,7 +1914,7 @@ def diff(arr, n: int, axis: int = 0):
out_arr[tuple(na_indexer)] = na

if arr.ndim == 2 and arr.dtype.name in _diff_special:
f = _diff_special[arr.dtype.name]
f = algos.diff_2d
f(arr, out_arr, n, axis)
else:
# To keep mypy happy, _res_indexer is a list while res_indexer is
Expand Down