Installation problem on FreeBSD 10.0 with Python 3.4.3 #10510

meeb · 2015-07-04T13:22:26Z

Hi,

I'm attempting to install pandas in a virtualenv on a FreeBSD system with Python 3.4.3. I can't see that I'm missing any obvious steps or dependancies. I get an error in a C module when importing pandas if I install pandas from pip, from a release tarball and with the repo (same error every time) that would imply something pretty major is missing. numpy, scipy and scikit-learn all install without any issues in the same venv. Example of the error:

(venv)hostname% pip install --upgrade pandas
Collecting pandas
Requirement already up-to-date: pytz>=2011k in ./venv/lib/python3.4/site-packages (from pandas)
Requirement already up-to-date: numpy>=1.7.0 in ./venv/lib/python3.4/site-packages (from pandas)
Requirement already up-to-date: python-dateutil>=2 in ./venv/lib/python3.4/site-packages (from pandas)
Requirement already up-to-date: six>=1.5 in ./venv/lib/python3.4/site-packages (from python-dateutil>=2->pandas)
Installing collected packages: pandas
Successfully installed pandas-0.16.2

(venv)hostname% python
Python 3.4.3 (default, Jun 18 2015, 17:00:30)
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
  File "/home/someuser/venv/lib/python3.4/site-packages/pandas/__init__.py", line 7, in <module>
    from pandas import hashtable, tslib, lib
ImportError: /home/someuser/venv/lib/python3.4/site-packages/pandas/hashtable.so: Undefined symbol "get_c_string"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/someuser/venv/lib/python3.4/site-packages/pandas/__init__.py", line 13, in <module>
    "extensions first.".format(module))
ImportError: C extension: /home/someuser/venv/lib/python3.4/site-packages/pandas/hashtable.so: Undefined symbol "get_c_string" not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace' to build the C extensions first.
>>>

When installing from a tarbal or the a checkout of the repo where setup.py is accessible and following the above suggestion (python setup.py build_ext --inplace) it compiles without issue it does not resolve the issue.

Some system information (print_versions.py taken from a repo checkout):

(venv)hostname% python print_versions.py

INSTALLED VERSIONS
------------------
commit: None
python: 3.4.3.final.0
python-bits: 64
OS: FreeBSD
OS-release: 10.1-RELEASE-p10
machine: amd64
processor: amd64
byteorder: little
LC_ALL: None
LANG: None

pandas: None
nose: 1.3.7
Cython: 0.22.1
numpy: 1.9.2
scipy: 0.15.1
statsmodels: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.4.2
pytz: 2015.4
bottleneck: 1.0.0
tables: None
numexpr: 2.4.3
matplotlib: 1.4.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext lo64)

Any pointers would be appreciated, thanks.

The text was updated successfully, but these errors were encountered:

bashtage · 2015-07-06T20:15:47Z

Are there errors related to Cython in the setup log if you run

python setup.py install

meeb · 2015-07-07T08:08:49Z

No errors from Cython. Cython was installed from pip. I have also tried removing Cython and installing/compiling it manually from the latest 0.22.1 tarball. Cython installs perfectly without issues as far as I can see.

I removed and re-installed Pandas from the repo after manually installing Cython and this had no effect. I have also been able to reproduce this error on a clean FreeBSD 10.1 vm.

bashtage · 2015-07-07T14:26:18Z

Sorry, I mean when you build install pandas. Something like:

cd ~/pandas
python setup.py clean
python setup.py build

Does the log from the last step show anything strange?

meeb · 2015-07-07T16:21:18Z

There's a load of warnings and notes, but it does seem to build all the required files. Here's the full output:

https://gist.githubusercontent.com/meeb/160fd9b409491cb9546a/raw/4dbad70b8f359a3714e82dd999571dfc0f471f28/gistfile1.txt

jreback · 2015-07-07T16:26:33Z

@meeb these are all ok

meeb · 2015-07-07T16:34:34Z

Yeah thought as much. Any other info I can provide or is this actually just something wrong in the build process on this platform?

jreback · 2015-07-07T16:37:18Z

I think you might have a partial pip install. try

pip uninstall pandas
pip install pandas

meeb · 2015-07-07T17:34:53Z

I've tried installing this in multiple completely clean venvs on different FreeBSD 10.1 installs (on different physical machines) all with the same results.

It's reproducible for Python3.4.2 on every FreeBSD 10.1 install I've tried so far. Interestingly, I've just noticed that installing pandas in a Python2.7.9 venv works fine without any errors at all on the same machines, both building from repo and installing via pip. Log:

https://gist.github.com/meeb/e68a685ad9acf038b7ae

iblislin · 2015-10-20T15:40:17Z

Hi!
I have python34 on FreeBSD 10.1, and face the same problem.
After digging around the code, I figure out that the root cause is the behavior of inline.
On gcc4x, it will choose gnu89 as the C model.

... But on FreeBSD 10, we are using clang3.4, which choose standard C99 model by default.
Ref is here

So... how to fix it? I will compile it with following command:

CFLAGS='-fgnu89-inline' python setup.py build_ext -f --inplace

Maybe the proper way is patching the Makefile.

jreback · 2015-10-20T17:06:15Z

way to do this properly is to put in options based on the platform in setup.py. Though AFAIK this has never been an issue before; I would make sure that cython is up to date.

iblislin · 2015-10-20T18:41:10Z

Hi @jreback !
Hmm... actually the problematic code do not come from cython. Those code from cython inline always carry static.
I will make some explanation.

E.g. consider pandas/src/numpy_helper.h

PANDAS_INLINE npy_int64                                                            
get_nat(void) {                                                                    
  return NPY_MIN_INT64;                                                            
}

Note that there is a macro called PANDAS_INLINE.

This macro is defined in pandas/src/helper.h.
By default, this macro will be expanded as __inline__, because clang will define __GUNC__ as well.

Those code are not generated by cython.

IMO, there are two solution for this inline problem. (To be honest, many project still have this problem on FreeBSD, e.g. our depend: ujson)

Simply using the CFLAGS -fgnu89-inline
Make sure our(also our depends) macro PANDAS_INLINE consider following situation:

__GNUC_GNU_INLINE__ (this is the behavior of --std=gnu89) define as __inline__
__GNUC_STDC_INLINE__ (--std=c99 or later) define as static inline

jreback · 2015-10-20T18:52:12Z

ok I see

well happy to take a patch that ifdefs things and add platform based CFLAGS is ok too (would add in setup.py)

jreback · 2015-10-20T18:53:06Z

btw tha pandas.json IS ujson (with a layer on top for pandas use)

iblislin · 2015-10-20T18:58:44Z

ok, i will send a PR later 😄
It's AM 3:00 here .... zzzzz

ghost · 2015-11-12T08:02:04Z

I know this issue is closed, but just for anyone interested: The problem also exists with Python 3.4.3 on FreeBSD 10.2 (freshly installed from CD) at least on my machine.

iblislin · 2015-11-12T10:00:53Z

Hi @usrbinerl !
Hmm... have you tried the patch in #11398?

ghost · 2015-11-12T10:34:53Z

Yes, I just did, but didn't resolve the issue. But take that with a grain of salt. I lack experience with compiling python from source, so I may have done something wrong.
So far I did not had to do that with python3 and my code works fine on Ubuntu 15.04/15.10 with pip3 installed pandas-package.

iblislin · 2015-11-12T10:54:41Z

@usrbinerl Here is my workflow, FYI :)

pyvenv-3.4 venv
source venv/bin/activate
pip install cython                          
pip install git+https://github.com/pydata/pandas.git

Note that cython first

Then, pandas works fine for me

(venv) > python
Python 3.4.3 (default, May 26 2015, 17:46:50) 
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>>

ghost · 2015-11-12T11:29:14Z

Thanks. I'll try with your tips again this afternoon.

jvkersch · 2016-01-24T05:44:19Z

@meeb @iblis17 @usrbinerl I encountered a similar problem on Mac OS (which also uses Clang) when building a Pandas with debug symbols. I tackled this by adding the static keyword to inline function declarations and in the process I removed the -fgnu89-inline flag that was added in 0ec93fb. Would you mind giving my branch #12123 a spin on your FreeBSD systems? Thanks!

iblislin · 2016-01-24T08:06:08Z

@jvkersch Nice work!
It works fine on my bsd box.

ghost · 2016-01-24T18:27:40Z

Works for me too now on FreeBSD. Thanks for your help.

It is currently impossible to build a debug version of Pandas on Mac OS, since some of the C function declarations are declared as `inline`, and compiling in debug mode turns off optimizations and hence inlining. While this could be solved by adding a compile argument to `setup.py` to use the C89 behavior for inline as in #10510, I believe an ultimately cleaner solution may consist of declaring inlined functions as `static`, so that the code is properly C99-compliant. For the implementation, I added the `static` keyword to the definitions of `PANDAS_INLINE` and `P_INLINE` (and removed duplicate occurrences of `static` from some function declarations), so that future implementations that use those defines are automatically compliant. This is slightly different from e.g. Cython, which defines `CYTHON_INLINE` as `inline` and declares everything as `static CYTHON_INLINE`. Author: Joris Vankerschaver <[email protected]> Closes #12123 from jvkersch/dev/static-inline and squashes the following commits: 6a225e7 [Joris Vankerschaver] MAINT: Remove compile flag that is no longer needed. a5138f0 [Joris Vankerschaver] DEV: Add static modifier to inline declarations.

bashtage · 2016-01-24T18:44:00Z

You might also want to try adding -std=c99 to the build, which uses modern, standard inlining behavior.

jreback · 2016-01-24T19:06:47Z

@bashtage is that a more general comment or only for FreeBSD?

meeb changed the title ~~installation problem on FreeBSD 10.0 with Python 3.4.3~~ Installation problem on FreeBSD 10.0 with Python 3.4.3 Jul 4, 2015

jreback added the Build Library building on various platforms label Jul 7, 2015

iblislin mentioned this issue Oct 21, 2015

BLD: add CFLAGS -fgnu89-inline on FreeBSD 10+ #11398

Merged

jreback added this to the 0.17.1 milestone Oct 21, 2015

iblislin added a commit to iblislin/pandas that referenced this issue Nov 10, 2015

BLD: Close pandas-dev#10510, add CFLAGS -fgnu89-inline on FreeBSD 10+

57426e1

jreback closed this as completed in #11398 Nov 10, 2015

jvkersch mentioned this issue Jan 24, 2016

DEV: Add static modifier to inline declarations. #12123

Closed

KetothXupack mentioned this issue Feb 25, 2016

error importing Pandas in python-3.4 unix environment - undefined symbol: pyobject_cmp #10608

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Installation problem on FreeBSD 10.0 with Python 3.4.3 #10510

Installation problem on FreeBSD 10.0 with Python 3.4.3 #10510

meeb commented Jul 4, 2015

bashtage commented Jul 6, 2015

meeb commented Jul 7, 2015

bashtage commented Jul 7, 2015

meeb commented Jul 7, 2015

jreback commented Jul 7, 2015

meeb commented Jul 7, 2015

jreback commented Jul 7, 2015

meeb commented Jul 7, 2015

iblislin commented Oct 20, 2015

jreback commented Oct 20, 2015

iblislin commented Oct 20, 2015

jreback commented Oct 20, 2015

jreback commented Oct 20, 2015

iblislin commented Oct 20, 2015

ghost commented Nov 12, 2015

iblislin commented Nov 12, 2015

ghost commented Nov 12, 2015

iblislin commented Nov 12, 2015

ghost commented Nov 12, 2015

jvkersch commented Jan 24, 2016

iblislin commented Jan 24, 2016

ghost commented Jan 24, 2016

bashtage commented Jan 24, 2016

jreback commented Jan 24, 2016

Installation problem on FreeBSD 10.0 with Python 3.4.3 #10510

Installation problem on FreeBSD 10.0 with Python 3.4.3 #10510

Comments

meeb commented Jul 4, 2015

bashtage commented Jul 6, 2015

meeb commented Jul 7, 2015

bashtage commented Jul 7, 2015

meeb commented Jul 7, 2015

jreback commented Jul 7, 2015

meeb commented Jul 7, 2015

jreback commented Jul 7, 2015

meeb commented Jul 7, 2015

iblislin commented Oct 20, 2015

jreback commented Oct 20, 2015

iblislin commented Oct 20, 2015

jreback commented Oct 20, 2015

jreback commented Oct 20, 2015

iblislin commented Oct 20, 2015

ghost commented Nov 12, 2015

iblislin commented Nov 12, 2015

ghost commented Nov 12, 2015

iblislin commented Nov 12, 2015

ghost commented Nov 12, 2015

jvkersch commented Jan 24, 2016

iblislin commented Jan 24, 2016

ghost commented Jan 24, 2016

bashtage commented Jan 24, 2016

jreback commented Jan 24, 2016