read_csv in multiple theads causes segmentation fault

[mrocklin]

The following script causes a segfault on my machine

from io import BytesIO
from multiprocessing.pool import ThreadPool
import pandas as pd

# Make many fake CSV files in memory
bytes = ['\n'.join(['%d,%d,%d' % (i,i,i) for i in range(10000)]).encode()
         for j in range(100)]
files = [BytesIO(b) for b in bytes]

# Read all files in many threads
pool = ThreadPool(8)
pool.map(pd.read_csv, files)

$ python script.py 
Segmentation fault (core dumped)

Python 3.4, Pandas 0.17.1, Ubuntu 14.04

[TomAugspurger]

FWIW on OSX that script just hangs at 99% CPU use. pandas 0.17.1, python 3.5.

[jreback]

cc @jdeschenes can you have a look

[jreback]

@mrocklin thanks for the repro!

[jdeschenes]

I think I found the issue, see my pull request. The issue was caused by a misplaced PyGilState_ensure(It was called after a Py_XDECREF being called.

Using read_csv with threads on such an object might have a big impact on performance.