BUG: pickle failing on FrozenList, when using MultiIndex (GH4788)

[jreback]

closes #4788

[jtratner]

@jreback did you see my comment? FrozenList never made it into a release, so we can go way simpler and just define a __reduce__ method...

[jreback]

yep...I just fixed it...much simpler that way....already done...now just going to regenerate the test pickles again....

[jtratner]

great :)

[jtratner]

and maybe we should leave a note to self to regenerate the legacy pickle right before we actually make the next release?

[jreback]

maybe....I only need to do it if I am actually changing the generate_legacy_pickles.py (which I did :)

in theory it should be stable (but in this case the writing of the pickle actually is different)

[jtratner]

why did you comment these out?

[jreback]

mistake....will update in a couple

[jreback]

hmm....py3 still give me an issue even with reduce.....let me see

[jtratner]

@jreback

Might be easier to just change this:

    def __reduce__(self):
        """Necessary for making this object picklable"""
        object_state = list(np.ndarray.__reduce__(self))
        subclass_state = (self.levels, self.labels, self.sortorder, self.names)
        object_state[2] = (object_state[2], subclass_state)
        return tuple(object_state)

to:

    def __reduce__(self):
        """Necessary for making this object picklable"""
        object_state = list(np.ndarray.__reduce__(self))
        subclass_state = (list(self.levels), list(self.labels), self.sortorder, list(self.names))
        object_state[2] = (object_state[2], subclass_state)
        return tuple(object_state)

and just not deal with having FrozenList in the pickle at all...

[jreback]

ok...will try that...set_state looks good then? (in core/index.py)

[jtratner]

let me look. I'm having trouble testing this locally... I get this error: ValueError: unsupported pickle protocol: 3

[jtratner]

actually, it's not quite right...one sec.

[jreback]

@jtratner ok...I updated with my hack it works now (I put in the changes to reduce by no dice)...if you can get it to work w/o the hacks great.....lmk.....I will add 32-bit stuff test pickles as welll

[jtratner]

will do -- the pickle error was just a weird thing on my end...

[jreback]

@jtratner alright current version passes travis!

I have a good selection of files....0.12: 2.7/3 for 64-bit linux/AMD, 0.12-dev: 2.7/3.2 for 32-bit/64-bit linux, 0.11 only 3.3

if you have any more arch..pls add, just run: pandas/io/tests/generate_legacy.py, but you have to copy it to the location or it will pick up the default system python (e.g. a build location)...then put the file in the appropriate folder and push...

[jtratner]

@jreback when I reverted just the compat/pickle_compat.py part of this commit, it still passed Travis... do we need the hack? (It worked locally for me too...) https://travis-ci.org/jtratner/pandas/builds/11175350

[jreback]

hmm
if that works great
maybe I had a test file that had a pickles frozen list but cleaned it up

let me try that

[jtratner]

this still does leave FrozenNDArray in there for the labels. Is that problematic? Otherwise we could have __reduce__() generate list of lists for labels and levels...

[jreback]

when I take out the hack, py3 fails on the 0.12-dev files (e.g. the current ones)....

[jtratner]

@jreback you know - I guess I thought that it just ran through pickles for the current version no matter what. Do I need to generate the data first?

[jreback]

@jtratner no...all the files are there. it runs thru all files in each of the directories, then compares to the data that is in the generate_legacy_pickles (which in theory should work as far back as one would want to go, e.g. its not dependent on compat type stuff for example)

[jtratner]

@jreback I mean, does it test that round-tripping with a pickle works in the current version? (i.e., it generates pickles for current version into a temporary file, runs them and then deletes them)

[jtratner]

i.e.,checks that no one checks in changes like the FrozenList change that causes pickles to stop working...

[jreback]

@jtratner no doesn't round trip test per se (maybe could add that)....maybe that is better for the current version I suppose

It checks all of the static files (which I had generated for the current version)

[jreback]

@jtratner I pushed a round-trip test (which passes), but fails if I remove the hack in pickle_compat....so something is still happenign with frozen list.....maybe it needs a setstate ?

[jtratner]

@jreback yeah, maybe. Sorry about the confusion there - i'll go figure it out now.

[jreback]

cool...lmk

[jtratner]

@jreback I can't reproduce your issue on 3.2, 3.3 or 2.7. In all of them, I can roundtrip a dataframe with a MultiIndex using to_pickle and read_pickle without any issue. For that __reduce__() to work, it means breaking any pickles from the time 033a932 was committed through to today (about the past month). But I don't think any of that was in an official release so that should be fine. Are you sure that you are removing all of the 0.12-dev pickles?

For example, if you make a pickle off of current master, then tried to use this, it would fail.

[jreback]

all is the files in 0.11/0.12/0.13 are brand new I regenerated them as part of this pr

I just call the geneate_legacy_pickles.py script with whatever version

[jtratner]

here's the passing Travis - https://travis-ci.org/jtratner/pandas/builds/11177267

[jreback]

hmm maybe I have an old pyc hanging out (that has the frozen list inside)

ok
I'll take it out
or u can

so u need set state in frozen list?

[jtratner]

nope, you don't need set_state. it just calls the first argument with the second argument. (I think).

So I'm pretty sure it basically it does (when reduce returns a 2-tuple):

f = FrozenList([1, 2, 3, 4])
reduced = f.__reduce__() # (FrozenList, ([1, 2, 3, 4],))
reduced[0](*reduced[1])

[jtratner]

Just to be clear why or whatnot:

class Something(object):
    def __init__(self, a, b):
        print("Something init called")
        self.a = a
        self.b = b
    def __repr__(self):
        return repr(self.__dict__)
    def __reduce__(self):
        return (Something, (self.a, self.b))

class Other(object):
    def __init__(self, a):
        print("Other Init called")
        self.a = a
        self.b = 2
    def __reduce__(self):
        return (Other, (self.a, 2))

import pickle
s = Something(1, 2)
r = pickle.dumps(s, protocol=-1)
s2 = pickle.loads(r)
print(s)
print(s2)
o = Other(1)
r = pickle.dumps(o, protocol=-1)
r2 = pickle.loads(r) # causes a TypeError because 'Other' called with 2 args

[jreback]

@jtratner I took the hack out and now works (I also took 0.13 fixed file testing out, now just a round-trip test is done).
No explicty testing on FrozenNDArray nor FrozenList, do we need?

[jtratner]

FrozenList shouldn't matter, because it shouldn't be ending up in pickles
anyways. If you're doing a MultiIndex, than you're already pickling
FrozenNDArray.

[jreback]

ok..ready to merge then?