CLN: Exception in pickle loading

[jbrockmendel]

• closes #xxxx
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[jbrockmendel]

@jorisvandenbossche @datapythonista since the CI failure is in web_and_docs. It looks like there is a problem trying to unpickle something from pytables? is this something we should have a regular test for?

[datapythonista]

The error is in the v0.8 release note I think. May be there is some code there doing things that are not valid anymore?

Otherwise adding a new tests sounds good.

[jorisvandenbossche]

Well, the docs is just code that is ran with the latest pandas. So there is a line of code that is working on master and failing with this branch (the pytables thing you see at the end is something else (ignored), you need to scroll up in the logs to find the actual error).

It's this (assuming 'pandas' is your local cloned repo):

In [7]: pd.read_pickle('pandas/doc/data/fx_prices') 
Out[7]: 
                EU      FR      GR      IT
DATE                                      
1980-01-01     NaN  4.0405  1.7246  804.74
1980-02-01     NaN  4.0963  1.7482  810.01
1980-03-01     NaN  4.3149  1.8519  860.13
1980-04-01     NaN  4.3536  1.8776  876.41
1980-05-01     NaN  4.1808  1.7913  843.23
...            ...     ...     ...     ...
2012-01-01  1.2910     NaN     NaN     NaN
2012-02-01  1.3238     NaN     NaN     NaN
2012-03-01  1.3208     NaN     NaN     NaN
2012-04-01  1.3160     NaN     NaN     NaN
2012-05-01  1.2806     NaN     NaN     NaN

[389 rows x 4 columns]

So we have a pickle file that reads fine on master, and the changes in this branch broke that.

Now, I assume the pickle file was written with pandas 0.8.0, so we don't guarantee support for that anymore, so it might not be a problem that you are breaking it (although t might be useful to understand why, to be sure it will not be a problem for more recent versions of pandas)

The error on this branch is:

>>>-------------------------------------------------------------------------
Exception in /home/vsts/work/1/s/doc/source/whatsnew/v0.8.0.rst at block ending on line None
Specify :okexcept: as an option in the ipython:: block to suppress this message
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-99c214b0f98c> in <module>
----> 1 fx = pd.read_pickle('data/fx_prices')

~/work/1/s/pandas/io/pickle.py in read_pickle(path, compression)
    156             # We want to silence any warnings about, e.g. moved modules.
    157             warnings.simplefilter("ignore", Warning)
--> 158             return pickle.load(f)
    159     except excs_to_catch:
    160         # e.g.

TypeError: _reconstruct: First argument must be a sub-type of ndarray
<<<-------------------------------------------------------------------------

[jreback]

can't remove that code in 0.8.0 whatsnew? (e.g. make it a code-block)?

[jbrockmendel]

looks like this worked

[jorisvandenbossche]

Note that by the changes in this PR, we now have a pickle file generated by pandas that we cannot longer read. So instead of just making this a code block, I would at least investigate a bit the reason why this happens to make sure this is not a valid use case that is now breaking.

[jreback]

Note that by the changes in this PR, we now have a pickle file generated by pandas that we cannot longer read. So instead of just making this a code block, I would at least investigate a bit the reason why this happens to make sure this is not a valid use case that is now breaking.

not sure I undertstand this last point. we no longer guarntee compatiblity for pickles < 0.20.3 (we did this in 0.25), so not sure this is an issue at all.

[jorisvandenbossche]

so not sure this is an issue at all.

I am not sure either. But if we are not sure, we should investigate it before just removing the code. The pickle file contains a dataframe with just float columns, so doesn't seem something that should break to me.

[jorisvandenbossche]

So since it was about encoding errors, I tried to write a simple pickle file with pandas 0.24.2 in Python 2, and read it with master and this branch:

In [1]: pd.__version__
Out[1]: u'0.24.2'

In [2]: df = pd.DataFrame(np.random.randn(10, 3), columns=['a', 'b', 'c'])

In [3]: df.to_pickle("test_py27.pkl")

and reading with master works, with this branch I get:

In [1]: pd.read_pickle("test_py27.pkl")
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-1-708cbd47a9cb> in <module>
----> 1 pd.read_pickle("test_py27.pkl")

~/scipy/pandas/pandas/io/pickle.py in read_pickle(path, compression)
    156             # We want to silence any warnings about, e.g. moved modules.
    157             warnings.simplefilter("ignore", Warning)
--> 158             return pickle.load(f)
    159     except excs_to_catch:
    160         # e.g.

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc6 in position 2: ordinal not in range(128)

[jbrockmendel]

@jorisvandenbossche excellent, can you make a test case out of this example? i'm fine with pinning it into this branch or seeing it go in first

[jbrockmendel]

updated with a test based on Joris's example

[jorisvandenbossche]

Looks good! small comment

[jorisvandenbossche]

Can you turn those again into ipython blocks now?

[jreback]

@jbrockmendel do we need to revert this?

[jbrockmendel]

this was reverted and then un-reverted, following #28645 (comment)

[jreback]

cool

[jorisvandenbossche]

@jbrockmendel the doc example still gives problems after restoring the py2 compat?
Note that it might not need to be fixed. If the error is now related to the fact that it is an old file (and not to the py2 unicode-decoding), that's fine to not fix (we do not have any guarantee to support pickle files from such old pandas versions)

[jbrockmendel]

the doc example still gives problems after restoring the py2 compat?

Yes. Re-disabling sounds goods.

[jbrockmendel]

re-disabled and green

[jorisvandenbossche]

@jreback

[jbrockmendel]

any outstanding issues here?