DEPR: GH10623 remove items from msgpack.encode for blocks

closes pandas-dev#10623
closes pandas-dev#12129

Author: kawochen

doc/source/io.rst

@@ -2539,6 +2539,24 @@ both on the writing (serialization), and reading (deserialization).
    optimizations in the io of the ``msgpack`` data. Since this is marked
    as an EXPERIMENTAL LIBRARY, the storage format may not be stable until a future release.
 
+   As a result of writing format changes and other issues:
+   +----------------------+------------------------+
+   | Packed with          | Can be unpacked with   |
+   +======================+========================+
+   | pre-0.17 / Python 2  | any                    |
+   +----------------------+------------------------+
+   | pre-0.17 / Python 3  | any                    |
+   +----------------------+------------------------+
+   | 0.17 / Python 2      | - 0.17 / Python 2      |
+   |                      | - >=0.18 / any Python  |
+   +----------------------+------------------------+
+   | 0.17 / Python 3      | >=0.18 / any Python    |
+   +----------------------+------------------------+
+   | 0.18                 | >= 0.18                |
+   +======================+========================+
+   
+   Reading (files packed by older versions) is backward-compatibile, except for files packed with 0.17 in Python 2, in which case only they can only be unpacked in Python 2.
+
 .. ipython:: python
 
    df = DataFrame(np.random.rand(5,2),columns=list('AB'))

doc/source/whatsnew/v0.18.0.txt

@@ -513,6 +513,33 @@ Subtraction by ``Timedelta`` in a ``Series`` by a ``Timestamp`` works (:issue:`1
 ``pd.Timestamp`` to rehydrate any timestamp like object from its isoformat
 (:issue:`12300`).
 
+Changes to msgpack
+^^^^^^^^^^^^^^^^^^
+
+Forward incompatible changes in ``msgpack`` writing format were made over 0.17.0 and 0.18.0; older versions of pandas cannot read files packed by newer versions (:issue:`12129`, `10527`)
+
+Bug in ``to_msgpack`` and ``read_msgpack`` introduced in 0.17.0 and fixed in 0.18.0, caused files packed in Python 2 unreadable by Python 3 (:issue:`12142`)
+
+.. warning::
+
+   As a result of a number of issues:
+
+   +----------------------+------------------------+
+   | Packed with          | Can be unpacked with   |
+   +======================+========================+
+   | pre-0.17 / Python 2  | any                    |
+   +----------------------+------------------------+
+   | pre-0.17 / Python 3  | any                    |
+   +----------------------+------------------------+
+   | 0.17 / Python 2      | - 0.17 / Python 2      |
+   |                      | - >=0.18 / any Python  |
+   +----------------------+------------------------+
+   | 0.17 / Python 3      | >=0.18 / any Python    |
+   +----------------------+------------------------+
+   | 0.18                 | >= 0.18                |
+   +======================+========================+
+
+   0.18.0 is backward-compatible for reading files packed by older versions, except for files packed with 0.17 in Python 2, in which case only they can only be unpacked in Python 2.
 
 Signature change for .rank
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -806,7 +833,6 @@ assignments are valid for multi-line expressions.
 
 Other API Changes
 ^^^^^^^^^^^^^^^^^
-
 - ``DataFrame.between_time`` and ``Series.between_time`` now only parse a fixed set of time strings. Parsing of date strings is no longer supported and raises a ``ValueError``. (:issue:`11818`)
 
   .. ipython:: python

pandas/core/common.py

@@ -3039,3 +3039,29 @@ def _random_state(state=None):
     else:
         raise ValueError("random_state must be an integer, a numpy "
                          "RandomState, or None")
+
+
+def pandas_dtype(dtype):
+    """
+    Converts input into a pandas only dtype object or a numpy dtype object.
+
+    Parameters
+    ----------
+    dtype : object to be converted
+
+    Returns
+    -------
+    np.dtype or a pandas dtype
+    """
+    if isinstance(dtype, compat.string_types):
+        try:
+            return DatetimeTZDtype.construct_from_string(dtype)
+        except TypeError:
+            pass
+
+        try:
+            return CategoricalDtype.construct_from_string(dtype)
+        except TypeError:
+            pass
+
+    return np.dtype(dtype)

pandas/core/internals.py

@@ -2098,6 +2098,14 @@ def __init__(self, values, placement, ndim=2, **kwargs):
 
         if not isinstance(values, self._holder):
             values = self._holder(values)
+
+        dtype = kwargs.pop('dtype', None)
+
+        if dtype is not None:
+            if isinstance(dtype, compat.string_types):
+                dtype = DatetimeTZDtype.construct_from_string(dtype)
+            values = values.tz_localize('UTC').tz_convert(dtype.tz)
+
         if values.tz is None:
             raise ValueError("cannot create a DatetimeTZBlock without a tz")
 
@@ -2428,6 +2436,10 @@ def make_block(values, placement, klass=None, ndim=None, dtype=None,
         else:
             klass = ObjectBlock
 
+    elif klass is DatetimeTZBlock and not is_datetimetz(values):
+        return klass(values, ndim=ndim, fastpath=fastpath,
+                     placement=placement, dtype=dtype)
+
     return klass(values, ndim=ndim, fastpath=fastpath, placement=placement)
 
 # TODO: flexible with index=None and/or items=None