ENH: use constant f32 eps, not np.finfo() during import

[jzwinck]

NumPy docs for np.finfo() say not to call it during import (at module scope).
It's a relatively expensive call, and it modifies the GIL state.
Now we just hard-code it, because it is always the value anyway.
This avoids touching the GIL at import, which helps avoid deadlocks in practice.

• closes import pandas hanging Flask 0.11.1 / Apache 2.4.18 #14641
• tests added / passed
• passes git diff upstream/master | flake8 --diff
• whatsnew entry

[jreback]

@jzwinck can you confirm that this 'solves' the issue (not asking for a test in the codebase as might be tricky). just some verification.

can you add a whatsnew note.

[codecov]

Codecov Report

Merging #15691 into master will decrease coverage by 0.02%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #15691      +/-   ##
==========================================
- Coverage   91.02%      91%   -0.03%     
==========================================
  Files         143      143              
  Lines       49409    49409              
==========================================
- Hits        44977    44967      -10     
- Misses       4432     4442      +10

Impacted Files	Coverage Δ
pandas/core/indexing.py	94.08% <100%> (ø)	✅
pandas/io/gbq.py	25% <0%> (-58.34%)	❌
pandas/tools/merge.py	93.68% <0%> (-0.34%)	❌
pandas/core/frame.py	97.87% <0%> (-0.1%)	❌
pandas/core/window.py	96.18% <0%> (ø)	✅
pandas/core/common.py	91.3% <0%> (+0.33%)	✅

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2cad4dd...dadb97c. Read the comment docs.

[jzwinck]

@jreback It "fixed" the hang on my system when I did import pandas in a new thread Python didn't know about and which did not hold the GIL. There could still be other ways to hang Python when the GIL is not used properly, but this patch sidesteps the one I encountered.

I have added a note in whatsnew 0.20.0 now.

As for testing, here is a C program which deadlocks before my patch and runs successfully after. Please note I believe it to be an erroneous program w.r.t. its multithreaded use of the Python C API--just to be clear this is not good code:

#include <Python.h>
#include <assert.h>
#include <pthread.h>

static void* import_pandas(void* arg)
{
    (void)arg;
    return PyImport_ImportModule("pandas");
}

int main(void)
{
    Py_Initialize();

    /* try to import pandas without the GIL
     * probably not a good idea, but... */
    pthread_t id;
    int rc = pthread_create(&id, 0, &import_pandas, 0);
    assert(rc == 0);

    /* if we time out, SIGALRM will kill us */
    alarm(10);

    void* obj;
    rc = pthread_join(id, &obj);
    assert(rc == 0);
    assert(obj);

    return 0;
}

If you want, you could add this as a unit test, but I'm not sure exactly how it fits in to the project given the existing tests all seem to be written in Python, and apart from using CFFI there's probably no way to implement the above in Python.

[jreback]

@jzwinck is there a way to do this directly using threading? IOW simulate importing inside the different threads.

[jzwinck]

@jreback I don't think so, because it requires misusing (or failing to use) the GIL, which no Python module short of FFI is going to facilitate.

…

On Mar 15, 2017 23:43, "Jeff Reback" ***@***.***> wrote: @jzwinck <https://github.com/jzwinck> is there a way to do this directly using threading? IOW simulate importing *inside* the different threads. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15691 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABKRF_4hRWJzG66wGF1DC6PSWtbdo33Pks5rmAcJgaJpZM4Md5gS> .

[jreback]

hmm, so how are that pure-python libraries that are simply doing imports failing? must be a race condition on the import itself in various threads.?

[jzwinck]

No pure Python program had this bug. All the reports in the linked issue are from Apache. My case was not Apache but was basically like the C test program I posted here. This issue is all about importing Pandas in an "embedded" Python interpreter.

…

On Mar 15, 2017 23:57, "Jeff Reback" ***@***.***> wrote: hmm, so how are that pure-python libraries that are simply doing imports failing? must be a race condition on the import itself in various threads.? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15691 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABKRFwd0hdAYfxV_x7rkNKnuBbd-9O6Sks5rmAp_gaJpZM4Md5gS> .

[jreback]

@jzwinck I c. ok will merge this and cross-fingers. thanks!