Skip to content

Release the GIL in Cython code #8882

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mrocklin opened this issue Nov 23, 2014 · 4 comments · Fixed by #10199
Closed

Release the GIL in Cython code #8882

mrocklin opened this issue Nov 23, 2014 · 4 comments · Fixed by #10199
Labels
Performance Memory or execution speed performance

Comments

@mrocklin
Copy link
Contributor

Pandas could release the GIL in parts of the Cython code. This would enable the parallel use of multiple cores with threading.

From off-line conversation with @jreback @cpcloud

http://lbolla.info/blog/2013/12/23/python-threads-cython-gil
@jreback jreback added the Performance Memory or execution speed performance label Nov 24, 2014
@jreback jreback added this to the 0.16.0 milestone Nov 24, 2014
@jreback jreback modified the milestones: 0.16.0, Next Major Release Mar 6, 2015
@TomAugspurger
Copy link
Contributor

FYI: scinkit-image just did this in scikit-image/scikit-image#1519.

@mrocklin
Copy link
Contributor Author

It happened surprisingly quickly in their case. I think that it was mostly the result of an afternoon sprint of a couple developers followed by a couple days of cleanup.

@jreback
Copy link
Contributor

jreback commented May 22, 2015

@cpcloud did some prelim work on this. I don't think its too difficult. Just requires some effort.

jreback added a commit to jreback/pandas that referenced this issue May 22, 2015
@jreback
wip on pandas-dev#8882, releasing the GIL
fe13de0
@jreback jreback mentioned this issue May 22, 2015
Merged
@shoyer
Copy link
Member

shoyer commented May 22, 2015

Indeed, it mostly just takes a little bit of testing work. Glad to see this being done!

For what it's worth, I looked into releasing the GIL a little bit in the hashtable/index API as part of my IntervalIndex work (#8707). The main stumbling block I encountered was pandas.hashtable.Int64Vector (and the other Vector objects). Hopefully someone more clever than me could figure out how to release the GIL on the append method by moving the resize call to use the C API: https://github.com/pydata/pandas/blob/0aceb38ac1a4659ae298d726bc447b97943163dc/pandas/hashtable.pyx#L68

jreback added a commit to jreback/pandas that referenced this issue May 26, 2015
@jreback
PERF: vbenches for pandas-dev#8882, releasing the GIL
ea32333
jreback added a commit to jreback/pandas that referenced this issue May 26, 2015
@jreback
wip on pandas-dev#8882, releasing the GIL
86fe4d6
jreback added a commit to jreback/pandas that referenced this issue May 28, 2015
@jreback
PERF: vbenches for pandas-dev#8882, releasing the GIL
08bb64a
jreback added a commit to jreback/pandas that referenced this issue May 28, 2015
@jreback
wip on pandas-dev#8882, releasing the GIL
5757201
jreback added a commit to jreback/pandas that referenced this issue May 28, 2015
@jreback
wip on pandas-dev#8882, releasing the GIL
2fe35db
jreback added a commit to jreback/pandas that referenced this issue Jun 3, 2015
@jreback
PERF: vbenches for pandas-dev#8882, releasing the GIL
502eb36
jreback added a commit to jreback/pandas that referenced this issue Jun 3, 2015
@jreback
wip on pandas-dev#8882, releasing the GIL
07fbe21
jreback added a commit to jreback/pandas that referenced this issue Jun 26, 2015
@jreback
PERF: vbenches for pandas-dev#8882, releasing the GIL
b08ab8e
jreback added a commit to jreback/pandas that referenced this issue Jun 26, 2015
@jreback
PERF: releasing the GIL, pandas-dev#8882
0bc2904
jreback added a commit that referenced this issue Jun 30, 2015
@jreback
Merge pull request #10199 from jreback/gil
16a44ad 
PERF: releasing the GIL, #8882
yarikoptic added a commit to neurodebian/pandas that referenced this issue Jul 2, 2015
@yarikoptic
Merge commit 'v0.16.2-65-g054821d' into debian
e360318 
* commit 'v0.16.2-65-g054821d':
  TST: 32-test fixes
  BUG: GH9907 generate_range when start and end have higher resolution than offset
  TST/CLN: remove assert_isinstance
  PERF: releasing the GIL, pandas-dev#8882
  PERF: vbenches for pandas-dev#8882, releasing the GIL
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Performance Memory or execution speed performance
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

PERF: releasing the GIL, #8882 jreback/pandas
4 participants
@mrocklin @jreback @shoyer @TomAugspurger