Test failure on Arm64

[bcdarwin]

Unfortunately I can't debug since I don't have such a machine set up; the tests were run by Hydra, the CI server for Nix/Nixpkgs for this PR.

Release 3.0.0.

FAIL: nibabel.tests.test_volumeutils.test_a2f_nan2zero_range
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/nix/store/iqysj9ccwz5xww339z07jyl6k5zc5074-python3.7-nose-1.3.7/lib/python3.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/build/nibabel-3.0.0/nibabel/tests/test_volumeutils.py", line 675, in test_a2f_nan2zero_range
    assert_array_equal([-128, -128, -128, nan_cast], back_arr)
  File "/nix/store/7k6dbm0p5imsnivp1xshbzkj30i28cjq-python3.7-numpy-1.18.0/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 936, in assert_array_equal
    verbose=verbose, header='Arrays are not equal')
  File "/nix/store/7k6dbm0p5imsnivp1xshbzkj30i28cjq-python3.7-numpy-1.18.0/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Arrays are not equal
Mismatched elements: 1 / 4 (25%)
Max absolute difference: 1
Max relative difference: 1.0
 x: array([-128, -128, -128,    0])
 y: array([-128, -128, -128,   -1], dtype=int8)
----------------------------------------------------------------------
Ran 7671 tests in 227.364s

[effigies]

This code is quite old and stable at this point, so my first inclination would be to look to see what's new in the environment.

Has Nix been packaging for Arm64 for a few versions, or is it just starting? It would be good to know if we're seeing this because now we're looking for it, or if something else has changed.

Assuming these tests aren't new, could we verify that this replicates with older versions of numpy?

[bcdarwin]

I can look into this tomorrow, but my first suspicion is that it's somehow related to long floats just as #860 was.

[matthew-brett]

It might well be something funny with long doubles. Can you put a guard in to print out what dt is, when the test errors?

[bcdarwin]

From examining the Hydra builds, the Arm64 build has been broken since January 2019, when we were building 2.3.1. It looks like this same test began failing with the upgrade to 2.3.3 in February 2019; the x64 build was successful during this period.

[bcdarwin]

@matthew-brett -- unfortunately I think debugging packages in this way is out of the scope of the Nixpkgs CI. If I have time I'll see about testing this on an Arm device.

[effigies]

I think Travis now has ARM64 builds. We should see about enabling one.

[matthew-brett]

An ARM64 build is an excellent idea.

[kaczmarj]

This probably isn't a good solution here because nibabel already has infrastructure in place to build wheels, but I wanted to include this for future readers. Docker has the ability to build and run images for different platforms, including arm64 and ppc64le. See https://www.docker.com/blog/getting-started-with-docker-for-arm-on-linux/.

Here are the steps to run nibabel on arm64 using Docker. I built an image with nibabel and pushed it to DockerHub.

# Register qemu interpreter things to be able to run arm64 etc on amd64
docker run --rm --privileged docker/binfmt:820fdd95a9972a5308930a2bdfb8573dd4447ad3

# Shell into an arm64 nibabel container (based on debian 10)
docker run --rm -it --workdir /tmp/for-testing --platform=linux/aarch64 kaczmarj/nibabel:latest
pytest --doctest-modules -v --pyargs nibabel.tests.test_volumeutils::test_a2f_nan2zero_range

Trimmed output:

    def test_a2f_nan2zero_range():
        # array_to_file should check if nan can be represented as zero
        # This comes about when the writer can't write the value (-intercept /
        # divslope) because it does not fit in the output range.  Input clipping
        # should not affect this
        fobj = BytesIO()
        # No problem for input integer types - they don't have NaNs
        for dt in INT_TYPES:
            arr_no_nan = np.array([-1, 0, 1, 2], dtype=dt)
            # No errors from explicit thresholding (nor for input float types)
            back_arr = write_return(arr_no_nan, fobj, np.int8, mn=1, nan2zero=True)
            assert_array_equal([1, 1, 1, 2], back_arr)
            back_arr = write_return(arr_no_nan, fobj, np.int8, mx=-1, nan2zero=True)
            assert_array_equal([-1, -1, -1, -1], back_arr)
            # Pushing zero outside the output data range does not generate error
            back_arr = write_return(arr_no_nan, fobj, np.int8, intercept=129, nan2zero=True)
            assert_array_equal([-128, -128, -128, -127], back_arr)
            back_arr = write_return(arr_no_nan, fobj, np.int8,
                                    intercept=257.1, divslope=2, nan2zero=True)
            assert_array_equal([-128, -128, -128, -128], back_arr)
        for dt in CFLOAT_TYPES:
            arr = np.array([-1, 0, 1, np.nan], dtype=dt)
            # Error occurs for arrays without nans too
            arr_no_nan = np.array([-1, 0, 1, 2], dtype=dt)
            # No errors from explicit thresholding
            # mn thresholding excluding zero
            assert_array_equal([1, 1, 1, 0],
                               write_return(arr, fobj, np.int8, mn=1))
            # mx thresholding excluding zero
            assert_array_equal([-1, -1, -1, 0],
                               write_return(arr, fobj, np.int8, mx=-1))
            # Errors from datatype threshold after scaling
            back_arr = write_return(arr, fobj, np.int8, intercept=128)
            assert_array_equal([-128, -128, -127, -128], back_arr)
            with pytest.raises(ValueError):
                write_return(arr, fobj, np.int8, intercept=129)
            with pytest.raises(ValueError):
                write_return(arr_no_nan, fobj, np.int8, intercept=129)
            # OK with nan2zero false, but we get whatever nan casts to
            nan_cast = np.array(np.nan).astype(np.int8)
            back_arr = write_return(arr, fobj, np.int8, intercept=129, nan2zero=False)
>           assert_array_equal([-128, -128, -128, nan_cast], back_arr)
E           AssertionError: 
E           Arrays are not equal
E           
E           Mismatch: 25%
E           Max absolute difference: 1
E           Max relative difference: 1.0
E            x: array([-128, -128, -128,    0])
E            y: array([-128, -128, -128,   -1], dtype=int8)

/src/nibabel/nibabel/tests/test_volumeutils.py:683: AssertionError

[kaczmarj]

I ran nibabel's tests on arm64, and only one failed.

FAILED tests/test_volumeutils.py::test_a2f_nan2zero_range - AssertionError: 
1 failed, 4784 passed, 111 skipped, 6 xfailed, 5662 warnings in 2153.20s (0:35:53)

To reproduce, run nibabel in an arm64 docker image:

# Register qemu interpreter things to be able to run arm64 etc on amd64
docker run --rm --privileged docker/binfmt:820fdd95a9972a5308930a2bdfb8573dd4447ad3
docker run --rm -it -w /tmp/for-testing --platform linux/aarch64 kaczmarj/nibabel:latest
pytest --doctest-modules -v --pyargs nibabel

The docker image includes numpy 1.16.2 and scipy 1.1.0. Here is the Dockerfile used to build the Docker image.

[bcdarwin]

This is the same test failure as I got on Hydra (Nixpkgs CI), so nice work creating a reprex that runs on x64.

[effigies]

Numpy got aarch64 wheels up on https://anaconda.org/scipy-wheels-nightly/numpy/files, so we're able to replicate this bug on Travis, now. (See #862.)

I'm not likely to have much time to fiddle with this until June, but others should feel completely welcome to take a stab at it.

[effigies]

Using @kaczmarj's image and running pytest --pdb nibabel/tests/test_volumeutils.py, I can at least narrow the case down to float128 on ARM64. Didn't have time to whittle it down to a minimal reproduction.

[effigies]

The issue turned out to be that np.float64(np.nan).astype(np.int8) is not the same as np.float128(np.nan).astype(np.int8) on Arm64.

The fix turns out to be quite simple: 5434b33

Would anybody care to review #862?

[effigies]

This should be resolved in master and will be fixed in the next bug-fix release.

@bcdarwin That can come quickly, if this will be useful to Arch. If you already have a workaround, then we can wait and see if any more bugs come up before a new release. Just let me know.

[bcdarwin]

Great! It's just a question of enabling Arm builds in the Nixpkgs repo. Since I doubt there are many Arm users yet we can probably wait until a new release. Meanwhile we could potentially pull in this commit as a patch.

[effigies]

I might wait until late June (after the OHBM hackathon) then, to see if anything comes up. But bug fix releases are cheap, so if it becomes useful to have an official version number with ARM support, just shout.