Skip to content

PERF: pd.Series.map too slow for huge dictionary #34717

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
charlesdong1991 opened this issue Jun 11, 2020 · 1 comment · Fixed by #34948
Closed

PERF: pd.Series.map too slow for huge dictionary #34717

charlesdong1991 opened this issue Jun 11, 2020 · 1 comment · Fixed by #34948
Labels
Performance Memory or execution speed performance
Milestone

Comments

@charlesdong1991
Copy link
Member

Just found out a performance issue with pd.Series.map, seems it is very slow when the input is a huge dictionary.

I noticed a similar issue reported before: #21278 and indeed for Series input, the first run might be slow and then for the later runs, they are very fast because hashable indexing is built. However, it doesn't seem to apply to dict input.

I slightly changed the example in #21278, and the runtime doesn't change if being run multiple times. And it is much faster using apply and dict.get.

So I am curious if this performance issue is being aware , and i would expect performance when a dict is assigned between pd.Series.map and pd.Series.apply(lambda x: blabla) is quite similar.

n = 1000000
domain = np.arange(0, n)
ranges = domain+10
maptable = pd.Series(ranges, index=domain).sort_index().to_dict()
query_vals = pd.Series([1,2,3])

%timeit query_vals.map(maptable)

while much faster if doing below:

query_vals.apply(lambda x: maptable.get(x))
@charlesdong1991 charlesdong1991 added Enhancement Needs Triage Issue that has not been reviewed by a pandas team member Performance Memory or execution speed performance and removed Enhancement Needs Triage Issue that has not been reviewed by a pandas team member labels Jun 11, 2020
@Rohith295
Copy link
Contributor

take

@jreback jreback added this to the Contributions Welcome milestone Jun 25, 2020
@jreback jreback modified the milestones: Contributions Welcome, 1.2 Sep 12, 2020
@Rohith295 Rohith295 removed their assignment Sep 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Performance Memory or execution speed performance
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants