BUG/CLN: Cleanup sanitize column and abstract broadcasting

[jtratner]

Also fixes small bug from #5321

[jreback]

I would move this test with the other duplicate method tests (even though its a long method), test_columns_dups_operations, or at least right after it

[jtratner]

Absolutely - was searching for 'duplicate' so I missed it :)

[jtratner]

@jreback A few points here:

1. This function never checked for column uniqueness previously, it always tried the lookup and then tested if it was a DataFrame or not. Thus it can't be a big performance hit (and my own perf testing shows that it isn't...)
2. It can't check for uniqueness because of issues with multilevel columns. E.g. consider this example:

df = pd.DataFrame(zip(range(-5,1), range(-5,1)), columns = pd.MultiIndex.from_arrays([['a', 'a'], ['x', 'y']]))
df['a'] = df.index

[jreback]

regarding your last comment....

yeh..perf may not be a problem....I am always wary of heavily called functions that have to deal with unique/non-unique...almost always I separate these cases (because of perf)

if its a multi-index it doesn't matter(so self.index.is_unique and not not isinstance(self.index,MultiIndex) should suffice (fyi....I would maybe add a method to do exactly this in your refactor (or could even override is_unique for mi, maybe)

[jtratner]

@jreback just thinking about it again, there's no reason to have a separate branch for Index, just needs to do an instance check for Index and ndarray first, then can use is_sequence.

[jtratner]

Is there a reason to prefer is_sequence() over checking np.isscalar(val)?

[jreback]

@jtratner that's fine...but then need to fix is_sequence (for the perf hit of iter, e.g. you can special case Index/Series in there I think``

you can try is_scalar....there is a prob a case where they differ (e.g. a Timestamp would hit is_sequence and not isscalar)

[jtratner]

@jreback wha? Timestamp is true for _is_sequence?

[jtratner]

ah, I see, is_scalar is False for timestamp, got it.

[jreback]

yep strings/Timestamp looks like lists and isscalr fails

[jreback]

@jtratner if perf checks out ok...this looks good!

[jtratner]

I ran the perf check 4 times on base commit and tip of this PR, everything came out about even on average, and the few that were particularly different were because 1 run of that particular benchmark took much more time than the other three, so I think it should be fine.

[jreback]

ok good to go then