Installation failed via pip on Mac OS X Python 3.9

[kengz]

Hi, I'm running into issues installing this via pip on Python 3.9, despite having few of the requirements already satisfied (see the first lines of the log below):

➜ pip install sklearn-pandas
Collecting sklearn-pandas
  Using cached sklearn_pandas-2.0.3-py2.py3-none-any.whl (10 kB)
Requirement already satisfied: numpy>=1.18.1 in /Users/xxx/miniconda3/envs/omni/lib/python3.9/site-packages (from sklearn-pandas) (1.19.2)
Requirement already satisfied: pandas>=1.0.5 in /Users/xxx/miniconda3/envs/omni/lib/python3.9/site-packages (from sklearn-pandas) (1.1.3)
Requirement already satisfied: scipy>=1.4.1 in /Users/xxx/miniconda3/envs/omni/lib/python3.9/site-packages (from sklearn-pandas) (1.5.2)
Requirement already satisfied: python-dateutil>=2.7.3 in /Users/xxx/miniconda3/envs/omni/lib/python3.9/site-packages (from pandas>=1.0.5->sklearn-pandas) (2.8.1)
Requirement already satisfied: numpy>=1.18.1 in /Users/xxx/miniconda3/envs/omni/lib/python3.9/site-packages (from sklearn-pandas) (1.19.2)
Requirement already satisfied: pytz>=2017.2 in /Users/xxx/miniconda3/envs/omni/lib/python3.9/site-packages (from pandas>=1.0.5->sklearn-pandas) (2020.4)
Requirement already satisfied: six>=1.5 in /Users/xxx/miniconda3/envs/omni/lib/python3.9/site-packages (from python-dateutil>=2.7.3->pandas>=1.0.5->sklearn-pandas) (1.15.0)
Collecting scikit-learn>=0.23.0
  Using cached scikit-learn-0.23.2.tar.gz (7.2 MB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/xxx/miniconda3/envs/omni/bin/python /Users/xxx/miniconda3/envs/omni/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/jx/z4vcr3393j537mmdc9jg1gsc0000gn/T/pip-build-env-vq1ymb5n/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.28.5' 'numpy==1.13.3; python_version=='"'"'3.6'"'"' and platform_system!='"'"'AIX'"'"' and platform_python_implementation == '"'"'CPython'"'"'' 'numpy==1.14.0; python_version=='"'"'3.6'"'"' and platform_system!='"'"'AIX'"'"' and platform_python_implementation != '"'"'CPython'"'"'' 'numpy==1.14.5; python_version=='"'"'3.7'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.17.3; python_version>='"'"'3.8'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.6'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.7'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.17.3; python_version>='"'"'3.8'"'"' and platform_system=='"'"'AIX'"'"'' 'scipy>=0.19.1'
       cwd: None
  Complete output (2494 lines):
  Ignoring numpy: markers 'python_version == "3.6" and platform_system != "AIX" and platform_python_implementation == "CPython"' don't match your environment
  Ignoring numpy: markers 'python_version == "3.6" and platform_system != "AIX" and platform_python_implementation != "CPython"' don't match your environment
  Ignoring numpy: markers 'python_version == "3.7" and platform_system != "AIX"' don't match your environment
  Ignoring numpy: markers 'python_version == "3.6" and platform_system == "AIX"' don't match your environment
  Ignoring numpy: markers 'python_version == "3.7" and platform_system == "AIX"' don't match your environment
  Ignoring numpy: markers 'python_version >= "3.8" and platform_system == "AIX"' don't match your environment
  Collecting Cython>=0.28.5
    Using cached Cython-0.29.21-py2.py3-none-any.whl (974 kB)
  Collecting numpy==1.17.3
    Using cached numpy-1.17.3.zip (6.4 MB)
  Collecting scipy>=0.19.1
    Using cached scipy-1.5.4-cp39-cp39-macosx_10_9_x86_64.whl (29.1 MB)
  Collecting setuptools
    Using cached setuptools-51.0.0-py3-none-any.whl (785 kB)
  Collecting wheel
    Using cached wheel-0.36.2-py2.py3-none-any.whl (35 kB)
  Building wheels for collected packages: numpy
    Building wheel for numpy (setup.py): started
    Building wheel for numpy (setup.py): finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /Users/xxx/miniconda3/envs/omni/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/jx/z4vcr3393j537mmdc9jg1gsc0000gn/T/pip-install-vpubhwxv/numpy_c9b5b84c35144de6b143a5317b9f20b6/setup.py'"'"'; __file__='"'"'/private/var/folders/jx/z4vcr3393j537mmdc9jg1gsc0000gn/T/pip-install-vpubhwxv/numpy_c9b5b84c35144de6b143a5317b9f20b6/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/jx/z4vcr3393j537mmdc9jg1gsc0000gn/T/pip-wheel-02k93ojp
         cwd: /private/var/folders/jx/z4vcr3393j537mmdc9jg1gsc0000gn/T/pip-install-vpubhwxv/numpy_c9b5b84c35144de6b143a5317b9f20b6/
    Complete output (2291 lines):
    Running from numpy source directory.
    blas_opt_info:
    blas_mkl_info:
    customize UnixCCompiler
      FOUND:
        libraries = ['mkl_rt', 'pthread']
        library_dirs = ['/Users/xxx/miniconda3/envs/omni/lib']
        define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
        include_dirs = ['/usr/local/include', '/Users/xxx/miniconda3/envs/omni/include']

      FOUND:
        libraries = ['mkl_rt', 'pthread']
        library_dirs = ['/Users/xxx/miniconda3/envs/omni/lib']
        define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
        include_dirs = ['/usr/local/include', '/Users/xxx/miniconda3/envs/omni/include']

    /bin/sh: svnversion: command not found
    non-existing path in 'numpy/distutils': 'site.cfg'
    lapack_opt_info:
    lapack_mkl_info:
    customize UnixCCompiler
      FOUND:
        libraries = ['mkl_rt', 'pthread']
        library_dirs = ['/Users/xxx/miniconda3/envs/omni/lib']
        define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
        include_dirs = ['/usr/local/include', '/Users/xxx/miniconda3/envs/omni/include']

      FOUND:
        libraries = ['mkl_rt', 'pthread']
        library_dirs = ['/Users/xxx/miniconda3/envs/omni/lib']
        define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
        include_dirs = ['/usr/local/include', '/Users/xxx/miniconda3/envs/omni/include']

    /Users/xxx/miniconda3/envs/omni/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'define_macros'
      warnings.warn(msg)
    running bdist_wheel
    running build
    running config_cc
    unifing config_cc, config, build_clib, build_ext, build commands --compiler options
    running config_fc
    unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
    running build_src
    build_src
    building py_modules sources
    creating build
    creating build/src.macosx-10.9-x86_64-3.9
    creating build/src.macosx-10.9-x86_64-3.9/numpy
    creating build/src.macosx-10.9-x86_64-3.9/numpy/distutils
    building library "npymath" sources
    get_default_fcompiler: matching types: '['gnu95', 'nag', 'absoft', 'ibm', 'intel', 'gnu', 'g95', 'pg']'
    customize Gnu95FCompiler
    Could not locate executable gfortran
    Could not locate executable f95
    customize NAGFCompiler
    customize AbsoftFCompiler
    Could not locate executable f90
    Could not locate executable f77
    customize IBMFCompiler
    Could not locate executable xlf90
    Could not locate executable xlf
    customize IntelFCompiler
    Could not locate executable ifort
    Could not locate executable ifc
    customize GnuFCompiler
    Could not locate executable g77
    customize G95FCompiler
    Could not locate executable g95
    customize PGroupFCompiler
    Could not locate executable pgfortran
    don't know how to compile Fortran code on platform 'posix'
    C compiler: clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/xxx/miniconda3/envs/omni/include -arch x86_64 -fPIC -O2 -isystem /Users/xxx/miniconda3/envs/omni/include -arch x86_64

    compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/Users/xxx/miniconda3/envs/omni/include/python3.9 -c'
    clang: _configtest.c
    clang -arch x86_64 _configtest.o -o _configtest
    success!
    removing: _configtest.c _configtest.o _configtest.o.d _configtest
    C compiler: clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/xxx/miniconda3/envs/omni/include -arch x86_64 -fPIC -O2 -isystem /Users/xxx/miniconda3/envs/omni/include -arch x86_64

    compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/Users/xxx/miniconda3/envs/omni/include/python3.9 -c'
    clang: _configtest.c
    _configtest.c:1:5: warning: incompatible redeclaration of library function 'exp' [-Wincompatible-library-redeclaration]
    int exp (void);
        ^
    _configtest.c:1:5: note: 'exp' is a builtin with type 'double (double)'
    1 warning generated.
    clang -arch x86_64 _configtest.o -o _configtest
    success!
    removing: _configtest.c _configtest.o _configtest.o.d _configtest
    creating build/src.macosx-10.9-x86_64-3.9/numpy/core
    creating build/src.macosx-10.9-x86_64-3.9/numpy/core/src
    creating build/src.macosx-10.9-x86_64-3.9/numpy/core/src/npymath
    conv_template:> build/src.macosx-10.9-x86_64-3.9/numpy/core/src/npymath/npy_math_internal.h
      adding 'build/src.macosx-10.9-x86_64-3.9/numpy/core/src/npymath' to include_dirs.
    conv_template:> build/src.macosx-10.9-x86_64-3.9/numpy/core/src/npymath/ieee754.c
    conv_template:> build/src.macosx-10.9-x86_64-3.9/numpy/core/src/npymath/npy_math_complex.c
    None - nothing done with h_files = ['build/src.macosx-10.9-x86_64-3.9/numpy/core/src/npymath/npy_math_internal.h']
    building library "npysort" sources
    creating build/src.macosx-10.9-x86_64-3.9/numpy/core/src/common
    conv_template:> build/src.macosx-10.9-x86_64-3.9/numpy/core/src/common/npy_sort.h
      adding 'build/src.macosx-10.9-x86_64-3.9/numpy/core/src/common' to include_dirs.
    creating build/src.macosx-10.9-x86_64-3.9/numpy/core/src/npysort
    conv_template:> build/src.macosx-10.9-x86_64-3.9/numpy/core/src/npysort/quicksort.c
    conv_template:> build/src.macosx-10.9-x86_64-3.9/numpy/core/src/npysort/mergesort.c
    conv_template:> build/src.macosx-10.9-x86_64-3.9/numpy/core/src/npysort/timsort.c
    conv_template:> build/src.macosx-10.9-x86_64-3.9/numpy/core/src/npysort/heapsort.c
    conv_template:> build/src.macosx-10.9-x86_64-3.9/numpy/core/src/npysort/radixsort.c
    conv_template:> build/src.macosx-10.9-x86_64-3.9/numpy/core/src/common/npy_partition.h
    conv_template:> build/src.macosx-10.9-x86_64-3.9/numpy/core/src/npysort/selection.c
    conv_template:> build/src.macosx-10.9-x86_64-3.9/numpy/core/src/common/npy_binsearch.h
    conv_template:> build/src.macosx-10.9-x86_64-3.9/numpy/core/src/npysort/binsearch.c
    None - nothing done with h_files = ['build/src.macosx-10.9-x86_64-3.9/numpy/core/src/common/npy_sort.h', 'build/src.macosx-10.9-x86_64-3.9/numpy/core/src/common/npy_partition.h', 'build/src.macosx-10.9-x86_64-3.9/numpy/core/src/common/npy_binsearch.h']
    building extension "numpy.core._dummy" sources
    Generating build/src.macosx-10.9-x86_64-3.9/numpy/core/include/numpy/config.h
    C compiler: clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/xxx/miniconda3/envs/omni/include -arch x86_64 -fPIC -O2 -isystem /Users/xxx/miniconda3/envs/omni/include -arch x86_64

    compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/Users/xxx/miniconda3/envs/omni/include/python3.9 -c'
    clang: _configtest.c
    success!
    removing: _configtest.c _configtest.o _configtest.o.d
    C compiler: clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/xxx/miniconda3/envs/omni/include -arch x86_64 -fPIC -O2 -isystem /Users/xxx/miniconda3/envs/omni/include -arch x86_64

    compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/Users/xxx/miniconda3/envs/omni/include/python3.9 -c'
    clang: _configtest.c
    _configtest.c:1:10: fatal error: 'endian.h' file not found
    #include <endian.h>
             ^~~~~~~~~~
    1 error generated.
    failure.
    removing: _configtest.c _configtest.o
    C compiler: clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/xxx/miniconda3/envs/omni/include -arch x86_64 -fPIC -O2 -isystem /Users/xxx/miniconda3/envs/omni/include -arch x86_64

    compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/Users/xxx/miniconda3/envs/omni/include/python3.9 -c'
    clang: _configtest.c
    _configtest.c:1:10: fatal error: 'sys/endian.h' file not found
    #include <sys/endian.h>
...

and the error log goes on for many pages and prohibits pasting here

[kengz]

Found workaround: manually install the dependencies first (scikit-learn, numpy, pandas) then installing this will not trigger dependency installation, and it only installs sklearn-pandas.
Still wonder why the dependency installation from sklearn-pandas fails

[ragrawal]

@kengz : thanks for reporting this. Python 3.9 is still not officially supported by sklearn-pandas. I can give it a try and see if we can start supporting it.

[ragrawal]

I am not able to reproduce the issue. Added Python 3.9 related tests. cases. Closing this for now.