Symbol not found: _floatify in pandas/lib.so on Mac 10.8 with numpy master #2188

cdeil · 2012-11-07T08:27:53Z

After updating pandas I can't import it any more because the symbol _floatify is undefined in pandas/lib.so:

$ python -c 'import pandas'
dlopen(/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so, 2): Symbol not found: _floatify
  Referenced from: /Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so
  Expected in: flat namespace
 in /Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/__init__.py", line 10, in <module>
    import pandas.lib as lib
ImportError: dlopen(/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so, 2): Symbol not found: _floatify
  Referenced from: /Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so
  Expected in: flat namespace
 in /Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so

$ otool -L /Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so
/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)
$ nm /Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so | grep floatify
                 U _floatify

The pandas build log doesn't show any problem with lib.so or floatify:
https://gist.github.com/4030153

My numpy install seems OK:
https://gist.github.com/4030177

The text was updated successfully, but these errors were encountered:

wesm · 2012-11-07T19:50:15Z

We had a little snafu in the setup.py file that was causing the Cython files to not automatically rebuild. The issue here was that floatify, a C function, had its signature changed but the declaration in the Cythonized C file had changed; hence the symbol lookup failure. At least that's what we think is going on. About to push some fixes (they should show up on here)-- do a clean build (setup.py clean, setup.py install) and let us know...

cc @yarikoptic

oops didn't mean to close issue

cdeil · 2012-11-07T21:58:58Z

With 5c945e1 I still see the same issue:
https://gist.github.com/4034766

wesm · 2012-11-07T22:17:52Z

What Cython version are you on? seems to be some kind of name mangling problem with clang going on. floatify is declared in pandas/src/util.pxd and the C function itself is in pandas/src/numpy_helper.h. I have no ability to debug this myself

cpcloud · 2012-11-07T23:13:24Z

Can you post the output of which python?

The reason I ask is because using the system Python can conflict with a Python from /opt/local/bin. I've definitely encountered a similar issue in the past.

cdeil · 2012-11-08T06:52:13Z

My Python and Cython are the latest Macports versions, my numpy is from git master:

$ xcodebuild -version
Xcode 4.5.2
Build version 4G2008a
$ which python
/opt/local/bin/python
$ python --version
Python 2.7.3
$ python -c 'import cython; print cython.__file__'
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/cython.pyc$ port installed py27-cython
The following ports are currently installed:
  py27-cython @0.17.1_0 (active)
$ python -c 'import numpy; print numpy.__file__; print numpy.__version__'
/Users/deil/Library/Python/2.7/lib/python/site-packages/numpy/__init__.pyc
1.8.0.dev-c5ccca9

Can you reproduce the issue with Cython 0.17.1?
If it's a name mangling problem, I could check which compilers were used for Python, numpy, pandas.
Or would it help if I post the generated c files that contain floatify?

dnmiller · 2012-11-08T07:08:52Z

Have you tried reinstalling Cython in /Users/deil/Library/Python/2.7 instead of the MacPorts dir? It seems risky to be grabbing libraries from two site-packages directories.

cdeil · 2012-11-08T08:08:28Z

@dnmiller Isn't having two site-packages normal?

There's packages that are hard to build with all their dependencies (like e.g. PyTables) which I install with a package manager (Macports on my Mac), and there's packages where I want the development version and in some cases even do development with, so these I install in my home folder.

I don't see how I could work with only one site-packages, because if I sudo install stuff myself in the site-packages of my package manager all hell breaks loose and I don't have the time to install / update all the packages I need myself.

I don't think it matters here where Cython is installed, rather I think it's a problem with the (Compiler, Python, Cython, numpy) combination I have.

dnmiller · 2012-11-08T09:07:59Z

Well, you should definitely check out the virtual environments package. It solves a lot of the problems you just described. But I agree that's not really the root cause.

While I can't say this is the same problem you're having, I've had problems with MacPorts linking to it's own versions of libc, libc++, liblapack, etc., which are incompatible with other programs compiled outside of MacPorts, which will link to non-MacPorts versions of the same libraries. I've seen a very similar error that was due to a non-MacPorts Python talking to a MacPorts version of NumPy.

My hunch is that the MacPorts Cython behaves differently than a version you would install in your home dir. You might consider trying to build pandas and its dependencies (and Cython) entirely in /Users/deil/Library/Python/2.7.

cpcloud · 2012-11-08T23:45:10Z

Have you tried manually importing another *.so file (just write a simple function in cython and try to import it)? If that doesn't work, it might hint at the problem being the clang linker flags, for which it seems there is a paucity of documentation, but I suspect that the fact that floatify is declared undefined in the binary might have something to do with your issue.

FYI: When I do nm lib.so | grep floatify I get T floatify as the output, but I'm running Arch Linux on a MacBook 4,2 so the code generation will probably be a bit different.

Also since you have the -g flag enabled when compiling the *.o files can you post the output of nm -l *.so | grep floatify? It should return the line number(s) of the function signature.

wesm · 2012-11-09T00:12:45Z

@fonnesbeck : sir, king of bleeding edge OS X technology, can you confirm that you can build and import pandas git master with your OS 10.8 rig?

brendam · 2012-11-09T02:21:26Z

I just tested in a new virtualenv on OS X 10.8.2 with latest xcode installed. I used 'pip install cython' to get cython. Everything else except wsgiref was installed by the pandas install.

clang --version 
Apple clang version 4.0 (tags/Apple/clang-421.0.60) (based on LLVM 3.1svn)
Target: x86_64-apple-darwin12.2.0
Thread model: posix

xcodebuild -version
Xcode 4.5.2
Build version 4G2008a

pip freeze
Cython==0.17.1
numpy==1.6.2
pandas==0.9.1.dev-0230bbf
python-dateutil==2.1
pytz==2012h
six==1.2.0
wsgiref==0.1.2

I used
python setup.py clean
python setup.py build_ext --inplace
python setup.py install
nosetests pandas

"Symbol not found: _floatify"

using nm lib.so | grep floatify I get U _floatify

wesm · 2012-11-09T04:05:24Z

Got a machine over here where I can reproduce this. Will try to figure out what's up

cpcloud · 2012-11-09T04:08:40Z

I'm super curious to know what the heck the issue is!
On Nov 8, 2012 9:21 PM, "Brenda Moon" [email protected] wrote:

I just tested in a new virtualenv on OS X 10.8.2 with latest xcode
installed. I used 'pip install cython' to get cython. Everything else
except wsgiref was installed by the pandas install.

clang --version
Apple clang version 4.0 (tags/Apple/clang-421.0.60) (based on LLVM 3.1svn)
Target: x86_64-apple-darwin12.2.0
Thread model: posix

xcodebuild -version
Xcode 4.5.2
Build version 4G2008a

pip freeze
Cython==0.17.1
numpy==1.6.2
pandas==0.9.1.dev-0230bbf
python-dateutil==2.1
pytz==2012h
six==1.2.0
wsgiref==0.1.2

I used
python setup.py clean
python setup.py build_ext --inplace
python setup.py install
nosetests pandas

"Symbol not found: _floatify"

using nm lib.so | grep floatify I get U _floatify

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2188#issuecomment-10213525.

#2188

wesm · 2012-11-09T16:11:28Z

Well, it basically clang has stricter semantics about inline functions:

http://clang.llvm.org/compatibility.html#inline

Making floatify and its dependent functions not inline gets rid of the linker errors. Still a bit of a mystery because there are numerous other equivalently-declared inline functions that don't have this problem. It evidently has something to do with optimization passes and floatify not actually being inlined by the compiler

git master should build now. pls let us know if not

cdeil · 2012-11-09T17:49:30Z

Works for me. Thanks!

ghost · 2012-11-09T18:13:46Z

@changhiskhan , I have a feeling cc099d6 broke #2029, since I started
having trouble with tox a few days ago.
Since inlining seems to be the culprit here, could this be rolled back or could you give
some details on the issues it fixes?
I'd hate to dive back in to python packaging if I can bribe, beg or fight my way out of it.

wesm · 2012-11-09T18:17:44Z

@y-p The problem is that I believe python setup.py cython had stopped working on systems with Cython installed.

ghost · 2012-11-09T18:19:53Z

grrr. I'm a continent away, but you yet may hear my shrieks of pain...

* commit 'v0.9.0-169-g5c945e1': BUG: fill_value not being applied for mixed type frame pandas-dev#2191 BUG: bug in setup.py with cython and sdist command class pandas-dev#2188 BLD: c scope decl wrong

jassinm · 2012-11-19T16:06:15Z

can't build on master 35dd6ac

mkvirtualenv pandasdev

pip freeze
Cython==0.17.1
distribute==0.6.28
ipython==0.13.1
matplotlib==1.2.0
numpy==1.6.2
pandas==0.9.2.dev-35dd6ac
python-dateutil==2.1
pytz==2012h
scipy==0.11.0
six==1.2.0
wsgiref==0.1.2

python setup.py build_ext --inplace
python setup.py install
nosetests pandas

(pandasdev)~/D/D/g/pandas git:master ❯❯❯ nosetests pandas
E
======================================================================
ERROR: Failure: ImportError (dlopen(/Users/locojay/Documents/Dev/github/pandas/pandas/lib.so, 2): Symbol not found: _floatify
  Referenced from: /Users/locojay/Documents/Dev/github/pandas/pandas/lib.so
  Expected in: flat namespace
 in /Users/locojay/Documents/Dev/github/pandas/pandas/lib.so)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/nose/loader.py", line 390, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/local/lib/python2.7/site-packages/nose/importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/local/lib/python2.7/site-packages/nose/importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/Users/locojay/Documents/Dev/github/pandas/pandas/__init__.py", line 10, in <module>
    import pandas.lib as lib
ImportError: dlopen(/Users/locojay/Documents/Dev/github/pandas/pandas/lib.so, 2): Symbol not found: _floatify
  Referenced from: /Users/locojay/Documents/Dev/github/pandas/pandas/lib.so
  Expected in: flat namespace
 in /Users/locojay/Documents/Dev/github/pandas/pandas/lib.so

----------------------------------------------------------------------
Ran 1 test in 0.001s

wesm · 2012-11-19T16:07:52Z

I broke master again. Can someone find out how to detect clang at compile time so I can conditionally take out the inline markers?

dnmiller · 2012-11-19T20:51:27Z

Adding a "clang" option to the top of numpy_helper.h should work.
Source: http://clang.llvm.org/docs/LanguageExtensions.html#builtinmacros

Though according to the clang docs, this is official C99 behavior anyway, so maybe the behavior of "STDC_VERSION" in the definition of PANDAS_INLINE isn't correct?

wesm · 2012-11-21T04:49:46Z

It should be working now, I'll need someone to verify because I don't have anything to test on right now

jassinm · 2012-11-21T05:05:47Z

_floatify is fixed but


python setup.py clean
python setup.py install

In [1]: import pandas as pd
dlopen(/Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/pandas-0.9.2.dev_61766ec-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so, 2): Symbol not found: _xstrtod
  Referenced from: /Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/pandas-0.9.2.dev_61766ec-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so
  Expected in: flat namespace
 in /Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/pandas-0.9.2.dev_61766ec-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-af55e7023913> in <module>()
----> 1 import pandas as pd

/Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/pandas-0.9.2.dev_61766ec-py2.7-macosx-10.8-x86_64.egg/pandas/__init__.py in <module>()
      8
      9 try:
---> 10     import pandas.lib as lib
     11 except Exception:  # pragma: no cover
     12     import sys

ImportError: dlopen(/Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/pandas-0.9.2.dev_61766ec-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so, 2): Symbol not found: _xstrtod
  Referenced from: /Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/pandas-0.9.2.dev_61766ec-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so
  Expected in: flat namespace
 in /Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/pandas-0.9.2.dev_61766ec-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so

for the time being i just remove the PANDAS_INLINE from xstrod in parse_helper.h, end_line in tokenizer

erg · 2012-11-22T19:46:12Z

I have the same problem as locojay.

modern:~ erg$ uname -a
Darwin modern.local 12.2.1 Darwin Kernel Version 12.2.1: Thu Oct 18 12:13:47 PDT 2012; root:xnu-2050.20.9~1/RELEASE_X86_64 x86_64

selasley · 2012-11-24T19:24:57Z

I added a forward declaration for end_line and removed P_INLINE from the forward declaration of xstrtod in tokenizer.c and now I can build and import pandas in python 2.7.3 in OS X 10.8.2.
Update: 0.9.2.dev-2600432 still has
// forward declaration
double P_INLINE xstrtod(const char *p...);
in tokenizer.c. I had to remove P_INLINE in order to build and import pandas

milkypostman · 2012-11-25T06:41:15Z

ditto

wesm · 2012-11-25T18:53:57Z

can someone let me know if git master builds right now? I made some changes per above.

milkypostman · 2012-11-25T20:08:14Z

It compiles for me but when I try to import pandas I get:

Traceback (most recent call last):
  File "./testpandas.py", line 5, in <module>
    from pandas import *
  File "/Users/dcurtis/Dropbox/Development/pandas/pandas/__init__.py", line 30, in <module>
    from pandas.io.parsers import (read_csv, read_table, read_clipboard,
  File "/Users/dcurtis/Dropbox/Development/pandas/pandas/io/parsers.py", line 22, in <module>
    import pandas._parser as _parser
ImportError: dlopen(/Users/dcurtis/Dropbox/Development/pandas/pandas/_parser.so, 2): Symbol not found: _xstrtod
  Referenced from: /Users/dcurtis/Dropbox/Development/pandas/pandas/_parser.so
  Expected in: flat namespace
 in /Users/dcurtis/Dropbox/Development/pandas/pandas/_parser.so

…sues #2188

wesm · 2012-11-25T20:18:09Z

Could you try again? I made xstrtod not inline

selasley · 2012-11-25T20:54:26Z

On Nov 25, 2012, at 3:18 PM, Wes McKinney [email protected] wrote:

Could you try again? I made xstrtod not inline

Version 0.9.2.dev-1d78502 builds and imports fine for me with python 2.7.3 under OS X 10.8.2, and all nosetests pass.

wesm · 2012-11-25T21:08:16Z

Thanks. I'll go ahead and close this for now

wesm closed this as completed Nov 7, 2012

wesm reopened this Nov 7, 2012

changhiskhan added a commit that referenced this issue Nov 7, 2012

BUG: bug in setup.py with cython and sdist command class #2188

cc099d6

changhiskhan added a commit that referenced this issue Nov 9, 2012

BUG: remove inlining for some functions to maintain clang compatibility

2a22b5b

#2188

wesm closed this as completed Nov 9, 2012

wesm reopened this Nov 19, 2012

wesm added a commit that referenced this issue Nov 25, 2012

ENH: use Vector classes in hash tables. attempt to fix clang build is…

1d78502

…sues #2188

wesm closed this as completed Nov 25, 2012

cpcloud mentioned this issue Mar 9, 2013

fails to compile with Cython 0.19 zeromq/pyzmq#339

Closed

mdmueller mentioned this issue Sep 13, 2014

Import of astropy.table.Table on OSX fails after merge of C-based io.ascii astropy/astropy#2938

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Symbol not found: _floatify in pandas/lib.so on Mac 10.8 with numpy master #2188

Symbol not found: _floatify in pandas/lib.so on Mac 10.8 with numpy master #2188

cdeil commented Nov 7, 2012

wesm commented Nov 7, 2012

cdeil commented Nov 7, 2012

wesm commented Nov 7, 2012

cpcloud commented Nov 7, 2012

cdeil commented Nov 8, 2012

dnmiller commented Nov 8, 2012

cdeil commented Nov 8, 2012

dnmiller commented Nov 8, 2012

cpcloud commented Nov 8, 2012

wesm commented Nov 9, 2012

brendam commented Nov 9, 2012

wesm commented Nov 9, 2012

cpcloud commented Nov 9, 2012

wesm commented Nov 9, 2012

cdeil commented Nov 9, 2012

ghost commented Nov 9, 2012

wesm commented Nov 9, 2012

ghost commented Nov 9, 2012

jassinm commented Nov 19, 2012

wesm commented Nov 19, 2012

dnmiller commented Nov 19, 2012

wesm commented Nov 21, 2012

jassinm commented Nov 21, 2012

erg commented Nov 22, 2012

selasley commented Nov 24, 2012

milkypostman commented Nov 25, 2012

wesm commented Nov 25, 2012

milkypostman commented Nov 25, 2012

wesm commented Nov 25, 2012

selasley commented Nov 25, 2012

wesm commented Nov 25, 2012

Symbol not found: _floatify in pandas/lib.so on Mac 10.8 with numpy master #2188

Symbol not found: _floatify in pandas/lib.so on Mac 10.8 with numpy master #2188

Comments

cdeil commented Nov 7, 2012

wesm commented Nov 7, 2012

cdeil commented Nov 7, 2012

wesm commented Nov 7, 2012

cpcloud commented Nov 7, 2012

cdeil commented Nov 8, 2012

dnmiller commented Nov 8, 2012

cdeil commented Nov 8, 2012

dnmiller commented Nov 8, 2012

cpcloud commented Nov 8, 2012

wesm commented Nov 9, 2012

brendam commented Nov 9, 2012

wesm commented Nov 9, 2012

cpcloud commented Nov 9, 2012

wesm commented Nov 9, 2012

cdeil commented Nov 9, 2012

ghost commented Nov 9, 2012

wesm commented Nov 9, 2012

ghost commented Nov 9, 2012

jassinm commented Nov 19, 2012

wesm commented Nov 19, 2012

dnmiller commented Nov 19, 2012

wesm commented Nov 21, 2012

jassinm commented Nov 21, 2012

erg commented Nov 22, 2012

selasley commented Nov 24, 2012

milkypostman commented Nov 25, 2012

wesm commented Nov 25, 2012

milkypostman commented Nov 25, 2012

wesm commented Nov 25, 2012

selasley commented Nov 25, 2012

wesm commented Nov 25, 2012