Skip to content

PERF: use StringHashTable for value_counts / duplicated with strings #14860

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
2 tasks
jreback opened this issue Dec 12, 2016 · 2 comments
Open
2 tasks

PERF: use StringHashTable for value_counts / duplicated with strings #14860

jreback opened this issue Dec 12, 2016 · 2 comments
Assignees
@lithomas1
Labels
Algos Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff duplicated duplicated, drop_duplicates Performance Memory or execution speed performance Strings String extension data type and string data

Comments

@jreback
Copy link
Contributor

jreback commented Dec 12, 2016

xref #14859

  • .value_counts()
  • .duplicated()
@jreback jreback added Difficulty Intermediate Performance Memory or execution speed performance Strings String extension data type and string data labels Dec 12, 2016
@jreback jreback added this to the Next Major Release milestone Dec 12, 2016
@jreback jreback modified the milestones: Next Major Release, Next Minor Release Mar 29, 2017
@jreback jreback mentioned this issue Mar 29, 2017
Closed
@jreback jreback modified the milestones: Interesting Issues, Next Major Release Nov 26, 2017
@simonjayhawkins simonjayhawkins added Algos Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff duplicated duplicated, drop_duplicates labels Jun 11, 2022
@simonjayhawkins simonjayhawkins mentioned this issue Jun 13, 2022
Open
@mroeschke mroeschke removed this from the Contributions Welcome milestone Oct 13, 2022
@jbrockmendel
Copy link
Member

@mroeschke do we already do this for string[python]? i assume string[pyarrow] has a separate more-performant implementation?

@mroeschke
Copy link
Member

From a quick skim of pandas/_libs/hashtable_func_helper.pxi.in looks like we don't use the StringHashTable yet, still the object one.

Yeah the string[pyarrow] one we want to avoid going through here I imagine

@lithomas1 lithomas1 self-assigned this Mar 4, 2023
@lithomas1 lithomas1 mentioned this issue Mar 22, 2023
Closed
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lithomas1 lithomas1

Labels
Algos Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff duplicated duplicated, drop_duplicates Performance Memory or execution speed performance Strings String extension data type and string data
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jreback @jbrockmendel @mroeschke @simonjayhawkins @lithomas1