REF: de-duplicate nested-dict handling in DataFrame construction

[jbrockmendel]

• closes #xxxx
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

This was the motivation for #41707. This will introduce some overhead, but will also prevent the behaviors from getting out of sync again.

[jreback]

oh i like this!

[jorisvandenbossche]

This might have caused a big slowdown in one of the benchmarks: https://pandas.pydata.org/speed/pandas/#frame_ctor.FromDicts.time_nested_dict_int64?python=3.8&Cython=0.29.21

@jbrockmendel can you verify this?

[jbrockmendel]

That looks plausible; noted in the OP we'd take a perf hit in these cases

[jorisvandenbossche]

Yes, but "some overhead" turns out to be a 6x slowdown.
So then having the cython implementation (and the added effort of ensuring it is kept in sync) might be worth it.

[jbrockmendel]

do you have anything in particular in mind? could revert until after the release

[jorisvandenbossche]

I didn't follow the original context, but IIUC, this PR didn't "fix" anything, correct? (like a bug) But it did remove a faster cython implementation to reduce maintenance cost in the idea the added overhead would only be small?

If that's the case, and the overhead actually turns out to be not so small (6x according to the benchmark), then it's a tradeoff we need to make: do we find the speed worth it to maintain the cython implementation.
If we say yes to that, then yes I suppose this means to revert the PR; but it would not be only until after the release (not sure what the point of that would be).

[jbrockmendel]

I didn't follow the original context, but IIUC, this PR didn't "fix" anything, correct? (like a bug) But it did remove a faster cython implementation to reduce maintenance cost in the idea the added overhead would only be small?

This context is that this follows #41707, which did fix a bug in the Series constructor that didn't affect DataFrame construction because they had separate implementations.

[jorisvandenbossche]

I opened #42248 to keep track of this regression.