BUG: DataFrame.equals should not care about block order (GH #9330)

[dsm054]

closes #9330 and another version of the same problem here on SO by canonicalizing the block order during an equals comparison.

Tested at the block manager level and above it at the frame level.

[jreback]

lgtm, pls add a release note!

[jreback]

worth adding that test that round trips to hdf (in the example issue)?

[dsm054]

Yeah, makes sense. [I'm hoping that using dtype.name sorts out the weird TypeError issue.] Where should that test go?

[jreback]

put in io/tests/test_pytables.py

yes I think u have to use dtype.name

the prob is (and maybe want to include a categorical)

is that a categorical dtype is not safe to compare to a numpy dtype (but the name is)

side issue is that if u have 2 cat blocks (cats are block separated, 1 cat per) thrn I sm not sure how they sort

[dsm054]

Urf. Getting the categorical stuff right is going to be a bit tricky.

Can I assume that NaN will never be an element in a category? I would have thought it would be disallowed entirely but I don't understand the output when you build one containing one.

[jreback]

@dsm054 you can have a nan in a category

In [5]: df = DataFrame({'A' : Series(list('aabbca')).astype('category',categories=['c','a','b',np.nan]),
   ...:                 'B' : Series(list('aabbca')).astype('category',categories=['c','a','b'])})

In [6]: df
Out[6]: 
   A  B
0  a  a
1  a  a
2  b  b
3  b  b
4  c  c
5  a  a

In [7]: df._data
Out[7]: 
BlockManager
Items: Index([u'A', u'B'], dtype='object')
Axis 1: Int64Index([0, 1, 2, 3, 4, 5], dtype='int64')
CategoricalBlock: slice(0, 1, 1), 1 x 6, dtype: category
CategoricalBlock: slice(1, 2, 1), 1 x 6, dtype: category

[dsm054]

Blek. What's the right way to handle putting the categorical blocks -- which won't coalesce even if the categories are the same -- in a canonical order?

[jreback]

so, I would compare all 'regular' blocks (e.g. non-Categorical/non-Sparse) like you are doing. But these need special handling.

I think you can order by the mgr block order. These map to how the it is represented in the actual DataFrame, and so will be the same. You could actually do this for all blocks I think.

In [4]: df._data.blocks[0].mgr_locs.as_array
Out[4]: array([0])

In [5]: df._data.blocks[1].mgr_locs.as_array
Out[5]: array([1])

[dsm054]

@jreback: okay, I've got a version which passes both my tests and the OP's original case, by sorting on a (block type name, mgr_locs) tuple. I don't understand how the mgr blocks work well enough to judge that side of things, unfortunately, so I don't know whether matching non-consolidated blocks can be in the wrong order even after this.

Currently passing everywhere except for what looks like an unrelated library build error; probably it'll work the next time it rebuilds.

[jreback]

hmm, can you add a test with 2 categoricals (to your mixed test case)

[dsm054]

Beyond "a:i8;b:category;c:category2;d:category2", # categories, do you mean? Two categoricals only?

[jreback]

sorry missed that part of the test

[jreback]

merged via e9179fe

thanks @dsm054
always quality code from you!