cross compiling libgit2-sys 0.7.6 fails with http-parser installed on the host

[Ratzzo]

The details of my system are as follows:

$ uname -a
Linux lprt0334 4.17.3-1-ARCH #1 SMP PREEMPT Tue Jun 26 04:42:36 UTC 2018 x86_64 GNU/Linux
$ rustc --version
rustc 1.29.0-nightly (12ed235ad 2018-07-18)
$ cargo --version
cargo 1.29.0-nightly (af9e40c26 2018-07-05)

When cross compiling to armv7 (armv7-unknown-linux-gnueabihf) and http-parser is installed on the main system the build fails. It seems CMake is not looking for http-parser on the target paths:

-- Found HTTP_Parser: /usr/include  
-- Found ZLIB: /home/ratzzo/.cargo/registry/src/jiasu.xzqcsaa.nyc.mn-1ecc6299db9ec823/libgit2-sys-0.7.6/target/armv7-unknown-linux-gnueabihf/debug/build/libz-sys-8e2f5c9991e5cf14/out/lib/libz.a (found version "1.2.11") 

The naive, and fastest workaround is uninstalling http-parser from the host system. I've been looking around for some environment variable that I can pass to CMake to build it with the bundled http-parser with no luck.

Here is the full log:

[ratzzo@lprt0334 libgit2-sys-0.7.6]$ cargo build --target=armv7-unknown-linux-gnueabihf

Compiling libgit2-sys v0.7.6 (file:///home/ratzzo/.cargo/registry/src/jiasu.xzqcsaa.nyc.mn-1ecc6299db9ec823/libgit2-sys-0.7.6)
error: failed to run custom build command for `libgit2-sys v0.7.6 (file:///home/ratzzo/.cargo/registry/src/jiasu.xzqcsaa.nyc.mn-1ecc6299db9ec823/libgit2-sys-0.7.6)`
process didn't exit successfully: `/home/ratzzo/.cargo/registry/src/jiasu.xzqcsaa.nyc.mn-1ecc6299db9ec823/libgit2-sys-0.7.6/target/debug/build/libgit2-sys-6b8093a83cac0a9e/build-script-build` (exit code: 101)
--- stdout
running: "cmake" "/home/ratzzo/.cargo/registry/src/jiasu.xzqcsaa.nyc.mn-1ecc6299db9ec823/libgit2-sys-0.7.6/libgit2" "-DUSE_SSH=OFF" "-DUSE_OPENSSL=OFF" "-DUSE_HTTPS=OFF" "-DCURL=OFF" "-DBUILD_SHARED_LIBS=OFF" "-DBUILD_CLAR=OFF" "-DCMAKE_INSTALL_PREFIX=/home/ratzzo/.cargo/registry/src/jiasu.xzqcsaa.nyc.mn-1ecc6299db9ec823/libgit2-sys-0.7.6/target/armv7-unknown-linux-gnueabihf/debug/build/libgit2-sys-814a38abe60290b9/out" "-DCMAKE_C_FLAGS= -ffunction-sections -fdata-sections -fPIC -march=native -march=armv7-a" "-DCMAKE_C_COMPILER=/usr/bin/arm-linux-gnueabihf-gcc" "-DCMAKE_CXX_FLAGS= -ffunction-sections -fdata-sections -fPIC -march=armv7-a" "-DCMAKE_CXX_COMPILER=/usr/bin/arm-linux-gnueabihf-g++" "-DCMAKE_BUILD_TYPE=Debug"
-- The C compiler identification is GNU 8.1.0
-- Check for working C compiler: /usr/bin/arm-linux-gnueabihf-gcc
-- Check for working C compiler: /usr/bin/arm-linux-gnueabihf-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Found PkgConfig: /usr/bin/pkg-config (found version "1.5.1") 
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Performing Test HAVE_STRUCT_STAT_ST_MTIM
-- Performing Test HAVE_STRUCT_STAT_ST_MTIM - Success
-- Performing Test HAVE_STRUCT_STAT_ST_MTIMESPEC
-- Performing Test HAVE_STRUCT_STAT_ST_MTIMESPEC - Failed
-- Performing Test HAVE_STRUCT_STAT_MTIME_NSEC
-- Performing Test HAVE_STRUCT_STAT_MTIME_NSEC - Failed
-- Performing Test HAVE_STRUCT_STAT_NSEC
-- Performing Test HAVE_STRUCT_STAT_NSEC - Success
-- Performing Test IS_WALL_SUPPORTED
-- Performing Test IS_WALL_SUPPORTED - Success
-- Performing Test IS_WEXTRA_SUPPORTED
-- Performing Test IS_WEXTRA_SUPPORTED - Success
-- Performing Test IS_WDOCUMENTATION_SUPPORTED
-- Performing Test IS_WDOCUMENTATION_SUPPORTED - Failed
-- Performing Test IS_WNO-MISSING-FIELD-INITIALIZERS_SUPPORTED
-- Performing Test IS_WNO-MISSING-FIELD-INITIALIZERS_SUPPORTED - Success
-- Performing Test IS_WSTRICT-ALIASING_SUPPORTED
-- Performing Test IS_WSTRICT-ALIASING_SUPPORTED - Success
-- Performing Test IS_WSTRICT-PROTOTYPES_SUPPORTED
-- Performing Test IS_WSTRICT-PROTOTYPES_SUPPORTED - Success
-- Performing Test IS_WDECLARATION-AFTER-STATEMENT_SUPPORTED
-- Performing Test IS_WDECLARATION-AFTER-STATEMENT_SUPPORTED - Success
-- Performing Test IS_WSHIFT-COUNT-OVERFLOW_SUPPORTED
-- Performing Test IS_WSHIFT-COUNT-OVERFLOW_SUPPORTED - Success
-- Performing Test IS_WNO-UNUSED-CONST-VARIABLE_SUPPORTED
-- Performing Test IS_WNO-UNUSED-CONST-VARIABLE_SUPPORTED - Success
-- Performing Test IS_WNO-UNUSED-FUNCTION_SUPPORTED
-- Performing Test IS_WNO-UNUSED-FUNCTION_SUPPORTED - Success
-- Looking for regcomp_l
-- Looking for regcomp_l - not found
-- Looking for futimens
-- Looking for futimens - found
-- Looking for qsort_r
-- Looking for qsort_r - found
-- Looking for qsort_s
-- Looking for qsort_s - not found
-- Looking for clock_gettime in rt
-- Looking for clock_gettime in rt - not found
-- Found HTTP_Parser: /usr/include  
-- Found ZLIB: /home/ratzzo/.cargo/registry/src/jiasu.xzqcsaa.nyc.mn-1ecc6299db9ec823/libgit2-sys-0.7.6/target/armv7-unknown-linux-gnueabihf/debug/build/libz-sys-8e2f5c9991e5cf14/out/lib/libz.a (found version "1.2.11") 
-- LIBSSH2 not found. Set CMAKE_PREFIX_PATH if it is installed outside of the default search path.
-- Enabled features:
 * threadsafe, threadsafe support
 * SHA, using CollisionDetection
 * http-parser, http-parser support
 * zlib, using system zlib

-- Disabled features:
 * debugpool, debug pool allocator
 * tracing, tracing support
 * cURL, cURL for HTTP proxy support
 * HTTPS, no support
 * SSH, SSH transport support
 * SPNEGO, SPNEGO authentication support
 * iconv, iconv encoding conversion support

-- Configuring done
-- Generating done
-- Build files have been written to: /home/ratzzo/.cargo/registry/src/jiasu.xzqcsaa.nyc.mn-1ecc6299db9ec823/libgit2-sys-0.7.6/target/armv7-unknown-linux-gnueabihf/debug/build/libgit2-sys-814a38abe60290b9/out/build
running: "cmake" "--build" "." "--target" "install" "--config" "Debug" "--"
Scanning dependencies of target git2internal
[  1%] Building C object src/CMakeFiles/git2internal.dir/annotated_commit.c.o
[  2%] Building C object src/CMakeFiles/git2internal.dir/apply.c.o
[  2%] Building C object src/CMakeFiles/git2internal.dir/attr.c.o
[  3%] Building C object src/CMakeFiles/git2internal.dir/attr_file.c.o
[  4%] Building C object src/CMakeFiles/git2internal.dir/attrcache.c.o
[  4%] Building C object src/CMakeFiles/git2internal.dir/blame.c.o
[  5%] Building C object src/CMakeFiles/git2internal.dir/blame_git.c.o
[  6%] Building C object src/CMakeFiles/git2internal.dir/blob.c.o
[  6%] Building C object src/CMakeFiles/git2internal.dir/branch.c.o
[  7%] Building C object src/CMakeFiles/git2internal.dir/buf_text.c.o
[  8%] Building C object src/CMakeFiles/git2internal.dir/buffer.c.o
[  8%] Building C object src/CMakeFiles/git2internal.dir/cache.c.o
[  9%] Building C object src/CMakeFiles/git2internal.dir/checkout.c.o
[ 10%] Building C object src/CMakeFiles/git2internal.dir/cherrypick.c.o
[ 10%] Building C object src/CMakeFiles/git2internal.dir/clone.c.o
[ 11%] Building C object src/CMakeFiles/git2internal.dir/commit.c.o
[ 12%] Building C object src/CMakeFiles/git2internal.dir/commit_list.c.o
[ 13%] Building C object src/CMakeFiles/git2internal.dir/config.c.o
[ 13%] Building C object src/CMakeFiles/git2internal.dir/config_cache.c.o
[ 14%] Building C object src/CMakeFiles/git2internal.dir/config_file.c.o
[ 15%] Building C object src/CMakeFiles/git2internal.dir/config_parse.c.o
[ 15%] Building C object src/CMakeFiles/git2internal.dir/crlf.c.o
[ 16%] Building C object src/CMakeFiles/git2internal.dir/date.c.o
[ 17%] Building C object src/CMakeFiles/git2internal.dir/delta.c.o
[ 17%] Building C object src/CMakeFiles/git2internal.dir/describe.c.o
[ 18%] Building C object src/CMakeFiles/git2internal.dir/diff.c.o
[ 19%] Building C object src/CMakeFiles/git2internal.dir/diff_driver.c.o
[ 19%] Building C object src/CMakeFiles/git2internal.dir/diff_file.c.o
[ 20%] Building C object src/CMakeFiles/git2internal.dir/diff_generate.c.o
[ 21%] Building C object src/CMakeFiles/git2internal.dir/diff_parse.c.o
[ 21%] Building C object src/CMakeFiles/git2internal.dir/diff_print.c.o
[ 22%] Building C object src/CMakeFiles/git2internal.dir/diff_stats.c.o
[ 23%] Building C object src/CMakeFiles/git2internal.dir/diff_tform.c.o
[ 23%] Building C object src/CMakeFiles/git2internal.dir/diff_xdiff.c.o
[ 24%] Building C object src/CMakeFiles/git2internal.dir/errors.c.o
[ 25%] Building C object src/CMakeFiles/git2internal.dir/fetch.c.o
[ 26%] Building C object src/CMakeFiles/git2internal.dir/fetchhead.c.o
[ 27%] Building C object src/CMakeFiles/git2internal.dir/fileops.c.o
[ 27%] Building C object src/CMakeFiles/git2internal.dir/filebuf.c.o
[ 28%] Building C object src/CMakeFiles/git2internal.dir/filter.c.o
[ 28%] Building C object src/CMakeFiles/git2internal.dir/fnmatch.c.o
[ 29%] Building C object src/CMakeFiles/git2internal.dir/global.c.o
[ 30%] Building C object src/CMakeFiles/git2internal.dir/graph.c.o
[ 30%] Building C object src/CMakeFiles/git2internal.dir/hash.c.o
[ 31%] Building C object src/CMakeFiles/git2internal.dir/hashsig.c.o
[ 32%] Building C object src/CMakeFiles/git2internal.dir/ident.c.o
[ 32%] Building C object src/CMakeFiles/git2internal.dir/idxmap.c.o
[ 33%] Building C object src/CMakeFiles/git2internal.dir/ignore.c.o
[ 34%] Building C object src/CMakeFiles/git2internal.dir/index.c.o
[ 34%] Building C object src/CMakeFiles/git2internal.dir/indexer.c.o
[ 35%] Building C object src/CMakeFiles/git2internal.dir/iterator.c.o
[ 36%] Building C object src/CMakeFiles/git2internal.dir/merge.c.o
[ 36%] Building C object src/CMakeFiles/git2internal.dir/merge_driver.c.o
[ 37%] Building C object src/CMakeFiles/git2internal.dir/merge_file.c.o
[ 38%] Building C object src/CMakeFiles/git2internal.dir/message.c.o
[ 39%] Building C object src/CMakeFiles/git2internal.dir/mwindow.c.o
[ 39%] Building C object src/CMakeFiles/git2internal.dir/netops.c.o
[ 40%] Building C object src/CMakeFiles/git2internal.dir/notes.c.o

--- stderr
fatal: not a git repository (or any of the parent directories): .git
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_CXX_COMPILER
    CMAKE_CXX_FLAGS
    USE_OPENSSL


/home/ratzzo/.cargo/registry/src/jiasu.xzqcsaa.nyc.mn-1ecc6299db9ec823/libgit2-sys-0.7.6/libgit2/src/netops.c:15:10: fatal error: http_parser.h: No such file or directory
 #include "http_parser.h"
          ^~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [src/CMakeFiles/git2internal.dir/build.make:791: src/CMakeFiles/git2internal.dir/netops.c.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:125: src/CMakeFiles/git2internal.dir/all] Error 2
make: *** [Makefile:130: all] Error 2
thread 'main' panicked at '
command did not execute successfully, got: exit code: 2

build script failed, must exit now', /home/ratzzo/.cargo/registry/src/jiasu.xzqcsaa.nyc.mn-1ecc6299db9ec823/cmake-0.1.31/src/lib.rs:643:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.

[alexcrichton]

Hm yeah that says -- Found HTTP_Parser: /usr/include which sounds like it's definitely wrong! I'm not sure how to force disable finding it though in libgit2's build system :(

This'll probably invole poking around libgit2's build system to see if there's a way we can force it http-parser to not be detected from the environment and the built-in one to be used

[Ratzzo]

It turns out http-parser doesn't provide a consistent way to find it, and there is a CMake module that doesn't seem to consider cross-compiling in libgit2: see libgit2/cmake/Modules/findHTTP_Parser.cmake

This'll probably invole poking around libgit2's build system to see if there's a way we can force it http-parser to not be detected from the environment and the built-in one to be used

There is this option in libgit2's CMakeLists.txt that can be used to force that:
OPTION( USE_EXT_HTTP_PARSER "Use system HTTP_Parser if available" ON)

Adding cfg.define("USE_EXT_HTTP_PARSER", "OFF"); to git2-rs/libgit2-sys/build.rs solved the issue

Maybe cross-compilation can be detected in git2-rs/libgit2-sys/build.rs, and only when cross compiling force it to use the bundled http-parser.

[alexcrichton]

Oh awesome, that sounds perfect! When the environment variable HOST is not equal to TARGET we're cross compiling, so seems reasonable to automatically pass this option. Want to send a PR for that?

[Ratzzo]

will do :)

[alexcrichton]

Should be fixed with https://github.com/alexcrichton/git2-rs/pull/338!