#30214 (Parallelized Build / CI) caused a build failure for me #30356
Comments
|
Not sure about Windows so if you have time / interest in taking a look that would be great. The PR reference actually allows parallel builds; before |
|
Yeah, I have the same problem in Windows and am now using just |
|
I'm not too familiar with building Cython and that traceback doesn't give many clues AFAICT. If this is a Windows problem, maybe then only pass the parallel arguments to build cython if we're not on windows? |
|
@scoder does the traceback here mean anything to you? |
jbrockmendel
added
Build
|
@jbrockmendel no, never seen this before. Probably specific to MS-Windows. The error message seems misleading – unless there is really a kind of "freezing" (py2exe etc.) going on on the user side, which I very much doubt. But it's some kind of multiprocessing hickup. |
|
if someone wanted to try adding the freeze_support call to setup.py might help
…Sent from my iPhone
On Dec 19, 2019, at 1:07 PM, Stefan Behnel ***@***.***> wrote:
@jbrockmendel no, never seen this before. Probably specific to MS-Windows. The error message seems misleading – unless there is really a kind of "freezing" (py2exe etc.) going on on the user side, which I very much doubt. But it's some kind of multiprocessing hickup.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
I tried adding |
|
I can maybe help debug.. I might try on a PR with the change to see if we can repro on our Azure windows jobs? |
|
@alimcmaster1 can you try upgrading Cython? I think this is a noop until 0.29.14 so you might not run into it on previous versions |
|
If it is helpful I believe in theory there is a way for the Windows compiler to perform parallel builds: https://docs.microsoft.com/en-us/cpp/build/reference/mp-build-with-multiple-processes?view=vs-2019 How distutils maps the Also note that in our own setup file we reuse the Which maybe could be the "culprit" for the issue above, and maybe that just needs to be decoupled for Windows users |
|
So from testing I do think it's the Line 523 in 1d36851 If this gets ignored for Windows I think the parallel build works, so could temporarily patch for now or see if there's an upstream fix for Cython @alimcmaster1 and/or @topper-123 are you seeing the same? |
|
Might be related to this issue on cython: cython/cython#3262 FWIW, I always used to use |
|
Thanks for confirming. Interested in a patch to not use the CLI argument as nthread for the Cython step on Windows? The parallel compilation thereafter could still help a lot
…Sent from my iPhone
On Dec 31, 2019, at 11:06 AM, Irv Lustig ***@***.***> wrote:
Might be related to this issue on cython: cython/cython#3262
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
Here's more. The way multiprocessing works in python you have to use I'll create a patch for Windows. |
Currently I can't get pandas build.
I get the message when running
python setup.py build_ext --inplace -j 4:I can't really parse what's going on, but the build seems to go back to the same RuntimeError repeatedly.
I've triaged the issue to stem from #30214. Any idea what's happening, @WillAyd ?
Workaround
I can work around the issue by building with parallelization, i.e. run
python setup.py build_ext --inplace -j 0instead ofpython setup.py build_ext --inplace -j 4.Output of
pd.show_versions()INSTALLED VERSIONS
commit : 95e1a63
python : 3.7.5.final.0
python-bits : 64
OS : Windows
OS-release : 10
machine : AMD64
processor : Intel64 Family 6 Model 78 Stepping 3, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None
pandas : 0.26.0.dev0+1358.g95e1a63dd
numpy : 1.17.4
pytz : 2019.1
dateutil : 2.8.0
pip : 19.3.1
setuptools : 42.0.2.post20191203
Cython : 0.29.13
pytest : 5.2.2
hypothesis : 4.28.2
sphinx : 2.2.0
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.10.3
IPython : 7.6.1
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.1.1
numexpr : 2.6.9
odfpy : None
openpyxl : 3.0.0
pandas_gbq : None
pyarrow : None
pytables : None
pytest : 5.2.2
s3fs : None
scipy : 1.3.1
sqlalchemy : None
tables : None
xarray : None
xlrd : 1.2.0
xlwt : None
xlsxwriter : None
The text was updated successfully, but these errors were encountered: