DataFrame constructor with dict of series misbehaving when columns specified

[TomAugspurger]

We take a strange path...

diff --git a/pandas/core/internals/construction.py b/pandas/core/internals/construction.py
index c43745679..54108ec58 100644
--- a/pandas/core/internals/construction.py
+++ b/pandas/core/internals/construction.py
@@ -552,6 +552,7 @@ def sanitize_array(data, index, dtype=None, copy=False,
             data = data.copy()
 
     # GH#846
+    import pdb; pdb.set_trace()
     if isinstance(data, (np.ndarray, Index, ABCSeries)):
 
         if dtype is not None:

In [1]: import pandas as pd; import numpy as np

In [2]: a = pd.Series(np.array([1, 2, 3]))
> /Users/taugspurger/sandbox/pandas/pandas/core/internals/construction.py(556)sanitize_array()
-> if isinstance(data, (np.ndarray, Index, ABCSeries)):
(Pdb) c

In [3]: r = pd.DataFrame({"A": a, "B": a}, columns=['A', 'B'])
> /Users/taugspurger/sandbox/pandas/pandas/core/internals/construction.py(556)sanitize_array()
-> if isinstance(data, (np.ndarray, Index, ABCSeries)):
(Pdb) up
> /Users/taugspurger/sandbox/pandas/pandas/core/series.py(259)__init__()
-> raise_cast_failure=True)
(Pdb)
> /Users/taugspurger/sandbox/pandas/pandas/core/series.py(301)_init_dict()
-> s = Series(values, index=keys, dtype=dtype)
(Pdb)
> /Users/taugspurger/sandbox/pandas/pandas/core/series.py(204)__init__()
-> data, index = self._init_dict(data, index, dtype)
(Pdb)
> /Users/taugspurger/sandbox/pandas/pandas/core/internals/construction.py(176)init_dict()
-> arrays = Series(data, index=columns, dtype=object)
(Pdb)
> /Users/taugspurger/sandbox/pandas/pandas/core/frame.py(387)__init__()
-> mgr = init_dict(data, index, columns, dtype=dtype)
(Pdb) data
{'A': 0    1
1    2
2    3
dtype: int64, 'B': 0    1
1    2
2    3
dtype: int64}
(Pdb) d
> /Users/taugspurger/sandbox/pandas/pandas/core/internals/construction.py(176)init_dict()
-> arrays = Series(data, index=columns, dtype=object)
(Pdb) l
171         Segregate Series based on type and coerce into matrices.
172         Needs to handle a lot of exceptional cases.
173         """
174         if columns is not None:
175             from pandas.core.series import Series
176  ->         arrays = Series(data, index=columns, dtype=object)
177             data_names = arrays.index
178
179             missing = arrays.isnull()
180             if index is None:
181                 # GH10856
(Pdb) data
{'A': 0    1
1    2
2    3
dtype: int64, 'B': 0    1
1    2
2    3
dtype: int64}

I'm guessing we don't want to be passing a dict of Series into the Series constructor there

[TomAugspurger]

We spend basically all our time in sanitize_array / _try_cast / construct_1d_object_array_from_listlike

And of that, all the time is in NumPy.

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
  1189                                           def construct_1d_object_array_from_listlike(values):
  1190                                               """
  1191                                               Transform any list-like object in a 1-dimensional numpy array of object
  1192                                               dtype.
  1193                                           
  1194                                               Parameters
  1195                                               ----------
  1196                                               values : any iterable which has a len()
  1197                                           
  1198                                               Raises
  1199                                               ------
  1200                                               TypeError
  1201                                                   * If `values` does not have a len()
  1202                                           
  1203                                               Returns
  1204                                               -------
  1205                                               1-dimensional numpy array of dtype object
  1206                                               """
  1207                                               # numpy will try to interpret nested lists as further dimensions, hence
  1208                                               # making a 1D array that contains list-likes is a bit tricky:
  1209         3         29.0      9.7      0.0      result = np.empty(len(values), dtype='object')
  1210         3     723782.0 241260.7    100.0      result[:] = values
  1211         3          3.0      1.0      0.0      return result

[misyntropy]

I would like to add my vote to actually go through with this performance enhancement (more like performance bug fix): constructing a DataFrame with index and columns as Index objects is around 200 times slower than constructing with np.zeros and then assigning to index and columns... see below. It's really weird and counter-intuitive given how (seemingly?) simple the operation is

df_a = pd.DataFrame(index=list(range(1000)))
df_b = pd.DataFrame(index=list(range(10000)))

%timeit df_c = pd.DataFrame(index=df_a.index,columns=df_b.index)
800 ms ± 70 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit df_d = pd.DataFrame(np.zeros((df_a.shape[0],df_b.shape[0]))); df_d.index = df_a.index; df_d.columns = df_b.index
4.36 ms ± 52 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

[TomAugspurger]

Yeah, we'll get there, probably for 0.24.1. Just need to do it in a way that's maintainable.