Skip to content

PERF/CLN: Use Numpy C Iterator API to unify isnaobj 1D/2D #50658

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
Jan 18, 2023
Merged

PERF/CLN: Use Numpy C Iterator API to unify isnaobj 1D/2D #50658

merged 9 commits into from
Jan 18, 2023

Conversation

lithomas1
Copy link
Member

@lithomas1 lithomas1 commented Jan 10, 2023 •
edited
Loading

  • closes #xxxx (Replace xxxx with the GitHub issue number)
  • Tests added and passed if fixing a bug or adding a new feature
  • All code checks passed.
  • Added type annotations to new arguments/methods/functions.
  • Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

ASV's

       before           after         ratio
     [d3995f76]       [aff44731]
     <main>           <clean-perf-isna>
-         104±5ms       31.0±0.3ms     0.30  frame_methods.Isnull.time_isnull_strngs
-         107±7ms         30.5±2ms     0.29  frame_methods.Isnull.time_isnull_obj

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

jbrockmendel
jbrockmendel reviewed Jan 10, 2023
View reviewed changes
pandas/_libs/missing.pyx Outdated
assert arr.ndim == 2, "'arr' must be 2-D."
bint is_null
ndarray result = np.empty((<object>arr).shape, dtype=np.uint8)
flatiter it = PyArray_IterNew(arr)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this robust to F-major inputs? i think we use the cnp.broadcast pattern for 2D support elsewhere

Copy link
Member Author

@lithomas1 lithomas1 Jan 10, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think PyArray_IterNew forces C style iteration.
https://numpy.org/devdocs/reference/c-api/array.html#c.PyArray_IterNew

I'll look to see if I need to add any tests for this.

I'll also give broadcasting a shot.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh neat. then it may turn out some places where ive used cnp.broadcast may be unnecessary

@mroeschke mroeschke added Internals Related to non-user accessible pandas implementation Clean labels Jan 12, 2023
mroeschke
mroeschke reviewed Jan 13, 2023
View reviewed changes
Copy link
Member

@mroeschke mroeschke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good just a merge conflict

@mroeschke
Copy link
Member 

mypy (stubtest)..........................................................Failed
- hook id: stubtest
- duration: 94.59s
- exit code: 1

pandas._libs.missing.isnaobj2d is not present at runtime

@mroeschke mroeschke added this to the 2.0 milestone Jan 18, 2023
@mroeschke mroeschke merged commit 9097263 into pandas-dev:main Jan 18, 2023
@mroeschke
Copy link
Member

Thanks @lithomas1

@lithomas1 lithomas1 deleted the clean-perf-isna branch January 23, 2023 01:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jbrockmendel jbrockmendel jbrockmendel left review comments

@mroeschke mroeschke mroeschke approved these changes

Assignees
No one assigned
Labels
Clean Internals Related to non-user accessible pandas implementation
Projects
None yet
Milestone
2.0
Development

Successfully merging this pull request may close these issues.
3 participants
@lithomas1 @mroeschke @jbrockmendel