CI Failing - Linux py37_np_dev - test_constructor_list_frames

[simonjayhawkins]

appeared on latest push to master, and recently updated PRs.

=================================== FAILURES ===================================
____________ TestDataFrameConstructors.test_constructor_list_frames ____________
[gw1] linux -- Python 3.7.3 /home/vsts/miniconda3/envs/pandas-dev/bin/python

self = <pandas.tests.frame.test_constructors.TestDataFrameConstructors object at 0x7fd3adeb3278>

    def test_constructor_list_frames(self):
        # see gh-3243
        result = DataFrame([DataFrame()])
>       assert result.shape == (1, 0)
E       assert (1, 1) == (1, 0)
E         At index 1 diff: 1 != 0
E         Use -v to get the full diff

pandas/tests/frame/test_constructors.py:119: AssertionError

[jreback]

hmm, something must have changed in numpy this has been stable forever

note that this should be the same

In [12]: DataFrame([np.array([])])
Out[12]: 
Empty DataFrame
Columns: []
Index: [0]

In [13]: DataFrame([np.array([])]).shape
Out[13]: (1, 0)

[jreback]

https://github.com/numpy/numpy/pull/13590/files or numpy/numpy#13507 is a recent change, so maybe the error that we catch has changed?

cc @mattip
cc @charris

[simonjayhawkins]

yeah.. the only change to the environment is numpy. (somewhere between and cf704e7f245e89c623bd82cbdba7c2dd07cf5fb4 and c9b25c939e0c64131b786a1922fe4c1f83db3e77) but that's 3 days of commits. checking to see if i can narrow it further.

[simonjayhawkins]

https://github.com/numpy/numpy/pull/13590/files or numpy/numpy#13507 is a recent change, so maybe the error that we catch has changed?

i think the error first appeared here. https://github.com/pandas-dev/pandas/runs/136802161 so before those changes

[simonjayhawkins]

@jreback

i've being trying to create a npdev conda (using anaconda) environment locally on Windows.

firstly the pip part does not seem to work

(base) C:\Users\simon\OneDrive\code\pandas-simonjayhawkins>conda env update -q --file=ci/deps/azure-37-numpydev.yaml --name npdev
Collecting package metadata: ...working... done
Solving environment: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

(base) C:\Users\simon\OneDrive\code\pandas-simonjayhawkins>conda env list

doing it manually for dateutil works

doing it manually for numpy and scripy...

(npdev) C:\Users\simon\OneDrive\code\pandas-simonjayhawkins>pip install -U -f https://7933911d6844c6c53a7d-47bd50c35cd79bd838daf386af554a83.ssl.cf2.rackcdn.com --pre numpy scipy
Looking in links: https://7933911d6844c6c53a7d-47bd50c35cd79bd838daf386af554a83.ssl.cf2.rackcdn.com
Collecting numpy
  Using cached https://7933911d6844c6c53a7d-47bd50c35cd79bd838daf386af554a83.ssl.cf2.rackcdn.com/numpy-1.17.0.dev0%2B20190507192239_3341590-cp37-cp37m-win_amd64.whl
Collecting scipy
  Using cached https://7933911d6844c6c53a7d-47bd50c35cd79bd838daf386af554a83.ssl.cf2.rackcdn.com/scipy-1.4.0.dev0%2B20190508181554_884216-cp37-cp37m-win_amd64.whl
Installing collected packages: numpy, scipy
  Found existing installation: numpy 1.17.0.dev0+3341590
    Uninstalling numpy-1.17.0.dev0+3341590:
      Successfully uninstalled numpy-1.17.0.dev0+3341590
  Found existing installation: scipy 1.4.0.dev0+884216
    Uninstalling scipy-1.4.0.dev0+884216:
      Successfully uninstalled scipy-1.4.0.dev0+884216
Successfully installed numpy-1.17.0.dev0+3341590 scipy-1.4.0.dev0+884216

numpy-1.17.0.dev0+3341590 is from 21 days ago.

so unless i'm doing something wrong. it looks like i wont be able to recreate the issue locally.

[jreback]

just install the wheel directly
the pip install instructions are on the ci files for that build

[mattip]

Could you give me a bit more context? If I build pandas HEAD against different versions of numpy, what should I be looking for?

[mattip]

(thanks for testing latest master HEAD by the way, it helps to flush these out before a release)

[charris]

@mattip I just hope it isn't something I backported. There should be wheels at http://wheels.scipy.org/ for the upcoming 1.16.4 release.

[TomAugspurger]

@mattip hopefully

pytest pandas/tests/frame/test_constructors.py -k test_constructor_list_frames

That fails for me locally, but I'm not immediately sure how to provide a Numpy-only example.

[simonjayhawkins]

just install the wheel directly

i think that numpy-1.17.0.dev0+20190507201151_3341590-cp35-cp35m-win_amd64.whl is the latest wheel built for the windows platform.

[charris]

@simonjayhawkins That's curious, looks like the recent builds for windows are not getting uploaded, the latest for mac and linux went up yesterday. In fact, it looks like the daily windows builds are not being done at all, the last one was due to a PR merge. @tylerjereddy @matthew-brett Thoughts?

EDIT: A PR merge to MacPython/numpy-wheels master.

[TomAugspurger]

@mattip not quite NumPy only yet, but here's a simpler example:

In [7]: import numpy as np; import pandas as pd

In [8]: np.array([pd.DataFrame()])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-8-af325d2f0a84> in <module>
----> 1 np.array([pd.DataFrame()])

ValueError: could not broadcast input array from shape (0,0) into shape (0)

previously (1.16.2, probably other versions) that was

In [43]: np.array([pd.DataFrame()])
Out[43]: array([], shape=(1, 0), dtype=float64)

[simonjayhawkins]

@mattip I just hope it isn't something I backported. There should be wheels at http://wheels.scipy.org/ for the upcoming 1.16.4 release.

thanks.. i'll be AFK for a few hours but if i'll check that build with pandas on Windows when i get back.

[tylerjereddy]

@charris Pretty sure cron job wheel builds are only activated for Travis (Linux / Mac) & not Appveyor (Windows) for both NumPy and SciPy wheels infrastructures.

Looks like Matti might try to bisect NumPy based on discussion above, so I won't duplicate that effort unless requested.

[charris]

@tylerjereddy True, I was just wondering if there was a way to activate it.

[mattip]

PR numpy/numpy#13399 seems to have caused this, merged two days ago. Make sense since it changed the way we make lists-of-objects into ndarrays. Now digging for a test we can put into our regression testing and a fix. Thanks for the heads up, hope it is not too complicated to fix