Pandas creates a large number of unnecessary threads #9394
Labels
Multithreading
Parallelism in pandas
Comments
|
To help narrow this down: do you see the same behavior if you just import |
|
pandas doesn't create a thread pool at all, but does create a single global thread local to handle some caching for date format parsing. It is typical to setup a thread pool at import time as this is a global resource. Since these threads are not actually doing anything unless the code for them takes over, how does this actually matter? how is this unclean? |
|
Numpy does create some BLAS threads by default, but if I set OMP_NUM_THREADS=1 and NUMEXPR_NUM_THREADS=1 I don't see extra threads: |
|
Apologies for the huge stacktrace below, but you can see that after setting the environment variables, at least some of the new threads are blosc-related stuff created by what looks like pandas.io: |
|
@jreback The particular use case that's causing me trouble is when I use a thread pinning tool like likwid-pin (https://code.google.com/p/likwid/). Likwid does round-robin allocation of threads onto CPU cores. But since there are a ton of extra threads created by Pandas, I have to try to trick likwid into putting the "real" threads on distinct cores while ignoring the Pandas threads. I use likwid as an example, but it would be just as much of a pain for any other general-purpose thread pinning tool. You could imagine other problems, like increased script run time due to thread creation / destruction when running a very large number of small python tasks. I think a simple solution would be to have Pandas take XXX_NUM_THREADS settings for all the multithreaded libraries it uses. A better solution would make Pandas not pull in all sorts of extra machinery when you just import a simple class (e.g., Timestamp) |
|
well as I said so |
jreback
added
Multithreading
|
Yeah, you're probably right that the solution of not importing the entire world when small classes are imported is the better way to go. |
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Simply importing pandas creates a huge number of threads:
Setting OMP_NUM_THREADS=1 and NUMEXPR_NUM_THREADS=1 reduces the number of threads created, but it looks like a bunch of blosc threads are still being created.
More seriously, simply importing a small pandas component also creates a ton of threads:
This is kind of wasteful, and it makes it difficult to optimize thread handling on multicore systems using thread pinning or other scheduling techniques.
The text was updated successfully, but these errors were encountered: