ENH: Allow read_pickle to accept file-like objects

[mar-ses]

Code Sample, a copy-pastable example if possible

# Your code here
import portalocker

with portalocker.Lock("/path/to/file.pickle", "rb+", timeout=60) as file:
    file.seek(0)        # not sure why I had this from before but it shouldn't cause an issue right?
    df = pd.read_pickle(file)

    # do stuff to df

    file.seek(0)
    file.truncate()   # I think here I was trying to erase the previous file just in case because of the read-mode
    file.to_pickle(file)

Alternatively, a more concise, self-contained, reproducible example:

import portalocker
import pandas as pd

df = pd.DataFrame({'a':[1,2,3], 'b':[1,2,3]})
path = 'test.pickle'
df.to_pickle(path)

with portalocker.Lock(path, "rb+", timeout=60) as file:
    file.seek(0)
    df = pd.read_pickle(file)

Replace portalocker.Lock with open(path, 'rb+') and you get the same thing.

Problem description

The above used to work. However, having updated to the latest pandas version recently (0.25.1), it now produces an error on the pd.read_pickle, and based on other issues, it will also produce an error on df.to_pickle() if it gets that far. Am I doing it wrong or should I do it differently, or it is just a bug?

The error traceback is:

    df = pd.read_pickle(file)
  File "/HOME/anaconda3/lib/python3.6/site-packages/pandas/io/pickle.py", line 145, in read_pickle
    f, fh = _get_handle(path, "rb", compression=compression, is_text=False)
  File "HOME/anaconda3/lib/python3.6/site-packages/pandas/io/common.py", line 392, in _get_handle
    raise ValueError(msg)
ValueError: Unrecognized compression type: infer

Expected Output

Output of pd.show_versions()

This didn't work as it couldn't determine the version for pip for some reason (I have pip, I'm not sure what's going on here), but the pandas version I'm running is: 0.25.1

On my other machine, which also has the error:

INSTALLED VERSIONS

commit : None
python : 3.6.8.final.0
python-bits : 64
OS : Linux
OS-release : 4.18.16-041816-generic
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : en_GB.UTF-8

pandas : 0.25.1
numpy : 1.14.3
pytz : 2018.4
dateutil : 2.7.3
pip : 19.1.1
setuptools : 39.1.0
Cython : 0.28.2
pytest : 3.5.1
hypothesis : None
sphinx : 1.7.4
blosc : None
feather : None
xlsxwriter : 1.0.4
lxml.etree : 4.2.1
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.10
IPython : 6.4.0
pandas_datareader: None
bs4 : 4.6.0
bottleneck : 1.2.1
fastparquet : None
gcsfs : None
lxml.etree : 4.2.1
matplotlib : 3.1.1
numexpr : 2.6.5
odfpy : None
openpyxl : 2.5.3
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : 1.1.0
sqlalchemy : 1.2.7
tables : 3.4.3
xarray : None
xlrd : 1.1.0
xlwt : 1.3.0
xlsxwriter : 1.0.4

[mroeschke]

We would need a self-contained, reproducible example http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

[mar-ses]

Alright, here is just the bare bones of it:

import portalocker
import pandas as pd

df = pd.DataFrame({'a':[1,2,3], 'b':[1,2,3]})
path = 'test.pickle'
df.to_pickle(path)

with portalocker.Lock(path, "rb+", timeout=60) as file:
    file.seek(0)
    df = pd.read_pickle(file)

I checked, it's giving me the error. Moreover, the portalocker isn't really important and doesn't change anything, the following also gives the same error:

with open(path, 'rb+') as file:
   ...:     file.seek(0)
   ...:     df = pd.read_pickle(file)

[asishm]

The docs for read_pickle state

path : str
File path where the pickled object will be loaded.

Maybe this could be an enhancement, but in terms of currently documented behavior, I think getting an error is expected. Although maybe the error thrown should be a bit more precise

[mroeschke]

Good catch @asishm. Currently read_pickle only supports string paths. It's a reasonable enhancement to support file object as well.

[jreback]

#5924 and #26237 related

[mar-ses]

It also worked before, so whatever it was doing before, could it be reimplemented?

[jreback]

It also worked before, so whatever it was doing before, could it be reimplemented?

how so? this was never implemented

[mar-ses]

To be honest, now that I know I'm surprised too, but my code only stopped working when I updated pandas.

[RoelantStegmann]

I have the same with s3 files: I used to open them with smart_open and then providing them to read_pickle. It doesn't work anymore.