Skip to content

[ArrowStringArray] PERF: use pa.compute.match_substring_regex for str.fullmatch if available #41332

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 11 commits into from
May 17, 2021
Merged

[ArrowStringArray] PERF: use pa.compute.match_substring_regex for str.fullmatch if available #41332

merged 11 commits into from
May 17, 2021

Conversation

simonjayhawkins
Copy link
Member

No description provided.

@simonjayhawkins simonjayhawkins added Performance Memory or execution speed performance Strings String extension data type and string data labels May 5, 2021
simonjayhawkins
simonjayhawkins commented May 5, 2021
View reviewed changes
pandas/core/arrays/string_arrow.py Outdated

def _str_match(
self, pat: str, case: bool = True, flags: int = 0, na: Scalar = None
):
if pa_version_under4p0:
return super()._str_match(pat, case, flags, na)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this may be quicker if using the fallback.... will need to check.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

broken this change off into #41487, slighter quicker than master, to bring the perf for object fallback back to before the pyarrow native implementation was added

simonjayhawkins
simonjayhawkins commented May 5, 2021
View reviewed changes
pandas/core/arrays/string_arrow.py
return super()._str_fullmatch(pat, case, flags, na)

if not pat.endswith("$") or pat.endswith("//$"):
pat = pat + "$"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we probably don't need the check but maybe faster... will need to check.

@simonjayhawkins
Copy link
Member Author 

[ 50.00%] ··· strings.Methods.time_fullmatch                                                                                                              ok
[ 50.00%] ··· ============== ==========
                  dtype                
              -------------- ----------
                   str        27.6±0ms 
                  string      22.1±0ms 
               arrow_string   2.35±0ms 
              ============== ==========

object fallback

[ 50.00%] ··· strings.Methods.time_fullmatch                                                                                                              ok
[ 50.00%] ··· ============== ==========
                  dtype                
              -------------- ----------
                   str        26.6±0ms 
                  string      22.1±0ms 
               arrow_string   27.0±0ms 
              ============== ==========

@simonjayhawkins simonjayhawkins marked this pull request as ready for review May 17, 2021 18:44
@simonjayhawkins simonjayhawkins added this to the 1.3 milestone May 17, 2021
@jreback jreback merged commit 671cf86 into pandas-dev:master May 17, 2021
@jreback
Copy link
Contributor

jreback commented May 17, 2021

thanks !

@simonjayhawkins simonjayhawkins deleted the _str_fullmatch branch May 18, 2021 08:41
TLouf pushed a commit to TLouf/pandas that referenced this pull request Jun 1, 2021
@simonjayhawkins @TLouf
[ArrowStringArray] PERF: use pa.compute.match_substring_regex for str…
04fd235 
….fullmatch if available (pandas-dev#41332)
JulianWgs pushed a commit to JulianWgs/pandas that referenced this pull request Jul 3, 2021
@simonjayhawkins @JulianWgs
[ArrowStringArray] PERF: use pa.compute.match_substring_regex for str…
ab0f649 
….fullmatch if available (pandas-dev#41332)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
Performance Memory or execution speed performance Strings String extension data type and string data
Projects
None yet
Milestone
1.3
Development

Successfully merging this pull request may close these issues.
2 participants
@simonjayhawkins @jreback