BUG: read_pickle fallback to latin_1 upon a UnicodeDecodeError

When a reading a pickle with MultiIndex columns generated in py27
`pickle_compat.load()` with `enconding=None` would throw an UnicodeDecodeError
when reading a pickle created in py27. Now, `read_pickle` catches that exception and
fallback to use `latin-1` explicitly.

Author: pedroreys

doc/source/whatsnew/v1.0.2.rst

@@ -38,6 +38,7 @@ Bug fixes
 
 - Using ``pd.NA`` with :meth:`DataFrame.to_json` now correctly outputs a null value instead of an empty object (:issue:`31615`)
 - Fixed bug in parquet roundtrip with nullable unsigned integer dtypes (:issue:`31896`).
+- Fixed bug where :meth:`pandas.io.pickleread_pickle` raised a ``UnicodeDecodeError` when reading a py27 pickle with MultiIndex column (:issue:`31988`).
 
 
 

pandas/io/pickle.py

@@ -183,7 +183,11 @@ def read_pickle(
         # e.g.
         #  "No module named 'pandas.core.sparse.series'"
         #  "Can't get attribute '__nat_unpickle' on <module 'pandas._libs.tslib"
-        return pc.load(f, encoding=None)
+        try:
+            return pc.load(f, encoding=None)
+        except UnicodeDecodeError:
+            # e.g. can occur for files written in py27; see GH#28645 and GH#31988
+            return pc.load(f, encoding="latin-1")
     except UnicodeDecodeError:
         # e.g. can occur for files written in py27; see GH#28645
         return pc.load(f, encoding="latin-1")

pandas/tests/io/data/pickle/test_mi_py27.pkl

@@ -501,3 +501,15 @@ def test_read_pickle_with_subclass():
 
     tm.assert_series_equal(result[0], expected[0])
     assert isinstance(result[1], MyTz)
+
+
+def test_read_py27_pickle_with_MultiIndex_column(datapath):
+    # pickle file with MultiIndex column written with py27
+    # should be readable without raising UnicodeDecodeError
+    # see GH#31988
+    path = datapath("io", "data", "pickle", "test_mi_py27.pkl")
+    df = pd.read_pickle(path)
+
+    # just test the columns are correct since the values are random
+    expected = pd.MultiIndex.from_arrays([["a", "b", "c"], ["A", "B", "C"]])
+    tm.assert_index_equal(df.columns, expected)