compat pickles don't have '_id'

[dalejung]

So I'm running into an issue with creating a Panel out of legacy DataFrame pickles. The issue is that _id is set to None and the current Index.is_ logic treats any two Indexes with unset _id as the same.

I thought the issue was with the unpickling. But it looks like there was an attempt to address this

def is_(self, other):
    return self._id is getattr(other, '_id', Ellipsis)

I'm guessing whoever thought that getattr(other, '_id', Ellipsis) would return Ellipsis if the _id was not set.

So I'm seeing:

1. Make unpickling make a new _id if it's not found in state
2. Make any Index.is_ always return False if either one has an unset id.

Maybe both?

[jreback]

you will have 2 provide a concrete example here of where this breaks and sample pickles

[jreback]

@dalejung ???? releasing 0.15.0 rc soon, If this is a problem would like to address before then.

can you should some code to reproduce?

[dalejung]

A self contained example is here:
http://nbviewer.ipython.org/gist/dalejung/e6d7fd95e2bef2cda81b
I attached the pickle files even though you can generate them yourself via dump.py.

The important part is pickling a legacy pandas object, the pickle api doesn't matter as _id isn't explicitly pickled either way and just comes along for the ride.

Man, it'd be nice if github could show ipython notebooks inline.

[jreback]

@dalejung your 0.12 unpickle just fine (for me)
the 0.14.1. have some weird error (unpickle protocol 4), so not sure what you are doing
I can unpickle 0.14.1 just fine.

So still unclear what actually doesn't work

[jreback]

these all read for me in py3.4 with the lastest version of master.

so if you have more info pls post.

[dalejung]

@jreback not sure we're on the same page. The issue isn't reading the pickle. The 0.12.0 pickles don't have the _id on their Indexes because that did not exist back then.

This is an issue because if

assert s1.index._id is None
assert s2.index._id is None

# then
s1.index.is_(s2.index) is True

So for something like this:

# indexes are not identical
assert s1.index.size != s2.index.size
df = pd.DataFrame({'s1' : s1, 's2' : s2}) # erros

/python/externals/pandas/pandas/core/internals.py in _stack_arrays(tuples, dtype)
   3697     stacked = np.empty(shape, dtype=dtype)
   3698     for i, arr in enumerate(arrays):
-> 3699         stacked[i] = _asarray_compat(arr)
   3700 
   3701     return stacked, placement

ValueError: could not broadcast input array from shape (27) into shape (30)

You get a shape mismatch because pandas doesn't union the two indexes because it already thinks they are equal. Normally it would've reindexed the data to the same shape.

[jreback]

@dalejung ok, easy enough. The identity was not being reset on legacy pickles. It didn't matter for current pickles because they go thru __new__ which already resets.

thanks for the report