Extremely slow construction of large DataFrames out of ndarrays?

[aldanor]

I was trying to figure out what's the most efficient way to create a dataframe out of a large numpy record array (I initially naively thought this could be done in a zero-copy way) and ran some simple tests:

---

n_columns = 2
n_records = int(15e6)

data = np.zeros((n_columns, n_records), np.int32)
arr = np.core.records.fromarrays(data)

%timeit -n1 -r1 df = pd.DataFrame(arr)
%timeit -n1 -r1 df = pd.DataFrame(arr, copy=False)
%timeit -n1 -r1 f0, f1 = arr['f0'].copy(), arr['f1'].copy()
f0, f1 = arr['f0'].copy(), arr['f1'].copy()
%timeit -n1 -r1 df = pd.DataFrame({'f0': f0, 'f1': f1})
%timeit -n1 -r1 df = pd.DataFrame({'f0': f0, 'f1': f1}, copy=False)

---

1 loops, best of 1: 2.47 s per loop
1 loops, best of 1: 2.42 s per loop
1 loops, best of 1: 48.2 ms per loop
1 loops, best of 1: 4.25 s per loop
1 loops, best of 1: 4.57 s per loop

---

I wonder what's the DataFrame constructor doing for several seconds straight even if it's provided with a materialized ndarray (note that copying both arrays takes < 50ms)? This is on v0.14.1.

P.S. Is there a more efficientway to make a dataframe out of a recarray?

[jreback]

use .from_record

what you are doing forces copies irregardless of the copy flag

[jreback]

sorry : from_records

[aldanor]

Ok, indeed, that seems more like it:

%timeit -n1 -r1 df = pd.DataFrame.from_records({'f0': f0, 'f1': f1})
%timeit -n1 -r1 df = pd.DataFrame.from_records(arr)

---

1 loops, best of 1: 46.5 ms per loop
1 loops, best of 1: 46.2 ms per loop

I just came across #4916 today and wrongly assumed that __init__ does the same as .from_records for recarrays. It's also quite unobvious that the constructor forces copying and ignores the copy flag in all the above cases (the last two in the first example in particular)... And why does it take 10x more time than it's needed to actually copy the ndarrays?

[jreback]

this is a longstanding API issue
pr is welcome (it's not really that hard actually just need a dispatch from the DataFrame.init constructor to actually call from_records essentially

[jreback]

closing this as it's a dupe (will note in the other issue)