CI Failing - Linux py37_np_dev - test_constructor_list_frames

[simonjayhawkins]

https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=29531&view=logs&j=3a03f79d-0b41-5610-1aa4-b4a014d0bc70&t=4d05ed0e-1ed3-5bff-dd63-1e957f2766a9

[mroeschke]

Looks like this build isn't available anymore. Happy to reopen if we continue to see this issue

[simonjayhawkins]

I'll leave this open since #32284 didn't fix the underlying issue. tests were skipped.

[simonjayhawkins]

this occured in 1.19.0rc1 (xref #34239) and tests are skipped again for 1.20.0.dev

This may become an issue shortly

cc @jbrockmendel

[TomAugspurger]

Here's the root cause

(Pdb) np.__version__
'1.18.5'
(Pdb) np.array([pd.DataFrame()])
array([], shape=(1, 0), dtype=float64)

(Pdb) np.__version__
'1.20.0.dev0+5345c25'
(Pdb) np.array([pd.DataFrame()])
array([], shape=(1, 0, 0), dtype=float64)

I think we should consider deprecating passing a list of DataFrame to the DataFrame constructor. We don't allow passing a list of 2D ndarrays, so we likely shouldn't allow passing a list of 2d DataFrames?

[jreback]

+1 on deprecating passing list of 2D things to the constructor.

[rebecca-palmer]

Nested DataFrames in general do still work, just not this method of creating them:

# df_outer=pd.DataFrame([df_inner,df_inner+10]) # fails in numpy 1.19
# mixed-type list works
df_outer=pd.DataFrame([8,df_inner,df_inner+10])
# or, setting the element later works
df_outer=pd.DataFrame([None,None])
df.outer.at[0,0]=df_inner

As noted above, it seems to happen because numpy.array now returns a 3D array that stacks the DataFrames, instead of a 1D array of objects.

The two failing tests were introduced in #3243 and #5324. We don't document anywhere obvious that nested DataFrames are supposed to work, but #5324 was reported by an actual user of this.

These two are not the only tests marked _is_numpy_dev, but are the only tests that fail with numpy 1.19 in Debian experimental's pandas 1.0.

The exception message it raises (pandas/core/internals/construction.py:324) could be changed to add a note about this issue.

[TomAugspurger]

This is failing the macpython builds now.

https://dev.azure.com/pandas-dev/pandas-wheels/_build/results?buildId=38048&view=logs&j=79f3a53a-4a6d-509c-98b5-ff6e9111f67c&t=25a63239-9c30-5cf0-cb4c-3d01da7b47c5&l=1049

___________ TestDataFrameConstructors.test_constructor_list_frames ____________
[gw0] win32 -- Python 3.6.8 D:\a\1\s\test_venv\Scripts\python.exe

self = <pandas.tests.frame.test_constructors.TestDataFrameConstructors object at 0x2882F090>

    @pytest.mark.xfail(_is_numpy_dev, reason="Interprets list of frame as 3D")
    def test_constructor_list_frames(self):
        # see gh-3243
>       result = DataFrame([DataFrame()])

test_venv\lib\site-packages\pandas\tests\frame\test_constructors.py:151: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
test_venv\lib\site-packages\pandas\core\frame.py:488: in __init__
    mgr = init_ndarray(data, index, columns, dtype=dtype, copy=copy)
test_venv\lib\site-packages\pandas\core\internals\construction.py:169: in init_ndarray
    values = prep_ndarray(values, copy=copy)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

values = array([], shape=(1, 0, 0), dtype=float64), copy = False

    def prep_ndarray(values, copy=True) -> np.ndarray:
        if not isinstance(values, (np.ndarray, ABCSeries, Index)):
            if len(values) == 0:
                return np.empty((0, 0), dtype=object)
            elif isinstance(values, range):
                arr = np.arange(values.start, values.stop, values.step, dtype="int64")
                return arr[..., np.newaxis]
    
            def convert(v):
                return maybe_convert_platform(v)
    
            # we could have a 1-dim or 2-dim list here
            # this is equiv of np.asarray, but does object conversion
            # and platform dtype preservation
            try:
                if is_list_like(values[0]) or hasattr(values[0], "len"):
                    values = np.array([convert(v) for v in values])
                elif isinstance(values[0], np.ndarray) and values[0].ndim == 0:
                    # GH#21861
                    values = np.array([convert(v) for v in values])
                else:
                    values = convert(values)
            except (ValueError, TypeError):
                values = convert(values)
    
        else:
    
            # drop subclass info, do not copy data
            values = np.asarray(values)
            if copy:
                values = values.copy()
    
>           raise ValueError("Must pass 2-d input")
E           ValueError: Must pass 2-d input

test_venv\lib\site-packages\pandas\core\internals\construction.py:295: ValueError

[TomAugspurger]

Looking at this again, I think we just remove the test, and perhaps add a whatsnew saying that because of upstream changes to NumPy this isn't supported anymore. DataFrame's are now consistently treated as 2D objects by NumPy, so we can't pretend that they're 1D in the constructor like we were.

old behavior

In [28]: np.__version__
Out[28]: '1.18.5'

In [29]: a = np.ones((0, 0))

In [30]: b = pd.DataFrame()

In [31]: np.array([a]).shape
Out[31]: (1, 0, 0)

In [32]: np.array([b]).shape
Out[32]: (1, 0)

In [33]: pd.DataFrame([a])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-33-49139b5fa086> in <module>
----> 1 pd.DataFrame([a])

~/miniconda3/envs/pandas=1.0.4/lib/python3.8/site-packages/pandas/core/frame.py in __init__(self, data, index, columns, dtype, copy)
    486                     mgr = arrays_to_mgr(arrays, columns, index, columns, dtype=dtype)
    487                 else:
--> 488                     mgr = init_ndarray(data, index, columns, dtype=dtype, copy=copy)
    489             else:
    490                 mgr = init_dict({}, index, columns, dtype=dtype)

~/miniconda3/envs/pandas=1.0.4/lib/python3.8/site-packages/pandas/core/internals/construction.py in init_ndarray(values, index, columns, dtype, copy)
    167     # by definition an array here
    168     # the dtypes will be coerced to a single dtype
--> 169     values = prep_ndarray(values, copy=copy)
    170
    171     if dtype is not None:

~/miniconda3/envs/pandas=1.0.4/lib/python3.8/site-packages/pandas/core/internals/construction.py in prep_ndarray(values, copy)
    293         values = values.reshape((values.shape[0], 1))
    294     elif values.ndim != 2:
--> 295         raise ValueError("Must pass 2-d input")
    296
    297     return values

ValueError: Must pass 2-d input

In [34]: pd.DataFrame([b])
Out[34]:
Empty DataFrame
Columns: []
Index: [0]

New behavior

In [4]: a = np.ones((0, 0))

In [5]: b = pd.DataFrame()

In [6]: np.array([a]).shape
Out[6]: (1, 0, 0)

In [7]: np.array([b]).shape
Out[7]: (1, 0, 0)

In [8]: pd.DataFrame([a])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-8-49139b5fa086> in <module>
----> 1 pd.DataFrame([a])

~/sandbox/pandas/pandas/core/frame.py in __init__(self, data, index, columns, dtype, copy)
    513                     mgr = arrays_to_mgr(arrays, columns, index, columns, dtype=dtype)
    514                 else:
--> 515                     mgr = init_ndarray(data, index, columns, dtype=dtype, copy=copy)
    516             else:
    517                 mgr = init_dict({}, index, columns, dtype=dtype)

~/sandbox/pandas/pandas/core/internals/construction.py in init_ndarray(values, index, columns, dtype, copy)
    188     # by definition an array here
    189     # the dtypes will be coerced to a single dtype
--> 190     values = _prep_ndarray(values, copy=copy)
    191
    192     if dtype is not None:

~/sandbox/pandas/pandas/core/internals/construction.py in _prep_ndarray(values, copy)
    322         values = values.reshape((values.shape[0], 1))
    323     elif values.ndim != 2:
--> 324         raise ValueError(f"Must pass 2-d input. shape={values.shape}")
    325
    326     return values

ValueError: Must pass 2-d input. shape=(1, 0, 0)

In [9]: pd.DataFrame([b])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-9-338077bdea83> in <module>
----> 1 pd.DataFrame([b])

~/sandbox/pandas/pandas/core/frame.py in __init__(self, data, index, columns, dtype, copy)
    513                     mgr = arrays_to_mgr(arrays, columns, index, columns, dtype=dtype)
    514                 else:
--> 515                     mgr = init_ndarray(data, index, columns, dtype=dtype, copy=copy)
    516             else:
    517                 mgr = init_dict({}, index, columns, dtype=dtype)

~/sandbox/pandas/pandas/core/internals/construction.py in init_ndarray(values, index, columns, dtype, copy)
    188     # by definition an array here
    189     # the dtypes will be coerced to a single dtype
--> 190     values = _prep_ndarray(values, copy=copy)
    191
    192     if dtype is not None:

~/sandbox/pandas/pandas/core/internals/construction.py in _prep_ndarray(values, copy)
    322         values = values.reshape((values.shape[0], 1))
    323     elif values.ndim != 2:
--> 324         raise ValueError(f"Must pass 2-d input. shape={values.shape}")
    325
    326     return values

ValueError: Must pass 2-d input. shape=(1, 0, 0)