DOC: added mentions in release notes, v0.11.1, basics

ENH: provide automatic list if multiple args passed to to_msgpack

DOC: changed docs to 0.12

ENH: iterator support for stream unpacking

Author: jreback

RELEASE.rst

@@ -32,6 +32,8 @@ pandas 0.11.1
 
   - pd.read_html() can now parse HTML string, files or urls and return dataframes
     courtesy of @cpcloud. (GH3477_)
+  - ``pd.read_msgpack()`` and ``pd.to_msgpack()`` are now a supported method of serialization
+    of arbitrary pandas (and python objects) in a lightweight portable binary format (GH686_)
 
 **Improvements to existing features**
 
@@ -75,6 +77,7 @@ pandas 0.11.1
 
 .. _GH3164: https://github.com/pydata/pandas/issues/3164
 .. _GH2786: https://github.com/pydata/pandas/issues/2786
+.. _GH686: https://github.com/pydata/pandas/issues/686
 .. _GH2194: https://github.com/pydata/pandas/issues/2194
 .. _GH3230: https://github.com/pydata/pandas/issues/3230
 .. _GH3251: https://github.com/pydata/pandas/issues/3251

doc/source/basics.rst

@@ -1192,47 +1192,6 @@ While float dtypes are unchanged.
    casted
    casted.dtypes
 
-.. _basics.serialize:
-
-Pickling and serialization
---------------------------
-
-All pandas objects are equipped with ``save`` methods which use Python's
-``cPickle`` module to save data structures to disk using the pickle format.
-
-.. ipython:: python
-
-   df
-   df.save('foo.pickle')
-
-The ``load`` function in the ``pandas`` namespace can be used to load any
-pickled pandas object (or any other pickled object) from file:
-
-
-.. ipython:: python
-
-   load('foo.pickle')
-
-There is also a ``save`` function which takes any object as its first argument:
-
-.. ipython:: python
-
-   save(df, 'foo.pickle')
-   load('foo.pickle')
-
-.. ipython:: python
-   :suppress:
-
-   import os
-   os.remove('foo.pickle')
-
-.. warning::
-
-   Loading pickled data received from untrusted sources can be unsafe.
-
-   See: http://docs.python.org/2.7/library/pickle.html
-
-
 Working with package options
 ----------------------------
 

doc/source/io.rst

@@ -981,6 +981,79 @@ one can use the ExcelWriter class, as in the following example:
 
 .. _io.hdf5:
 
+.. _basics.serialize:
+
+Serialization
+-------------
+
+msgpack
+~~~~~~~
+
+Starting in 0.12.0, pandas is supporting the ``msgpack`` format for 
+object serialization. This is a lightweight portable binary format, similar
+to binary JSON, that is highly space efficient, and provides good performance 
+both on the writing (serialization), and reading (deserialization).
+
+.. ipython:: python
+
+   df = DataFrame(np.random.rand(5,2),columns=list('AB'))
+   df.to_msgpack('foo.msg')
+   pd.read_msgpack('foo.msg')
+   s = Series(np.random.rand(5),index=date_range('20130101',periods=5))
+
+You can pass a list of objects and you will receive them back on deserialization.
+
+.. ipython:: python
+
+   pd.to_msgpack('foo.msg', df, 'foo', np.array([1,2,3]), s)
+   pd.read_msgpack('foo.msg')
+
+.. ipython:: python
+   :suppress:
+   :okexcept:
+
+   os.remove('foo.msg')
+
+
+pickling
+~~~~~~~~
+
+All pandas objects are equipped with ``save`` methods which use Python's
+``cPickle`` module to save data structures to disk using the pickle format.
+
+.. ipython:: python
+
+   df
+   df.save('foo.pickle')
+
+The ``load`` function in the ``pandas`` namespace can be used to load any
+pickled pandas object (or any other pickled object) from file:
+
+
+.. ipython:: python
+
+   load('foo.pickle')
+
+There is also a ``save`` function which takes any object as its first argument:
+
+.. ipython:: python
+
+   save(df, 'foo.pickle')
+   load('foo.pickle')
+
+.. ipython:: python
+   :suppress:
+
+   import os
+   os.remove('foo.pickle')
+
+.. warning::
+
+   Loading pickled data received from untrusted sources can be unsafe.
+
+   See: http://docs.python.org/2.7/library/pickle.html
+
+
 HDF5 (PyTables)
 ---------------
 

doc/source/v0.11.1.txt

@@ -1,18 +1,17 @@
-.. _whatsnew_0120:
+.. _whatsnew_0111:
 
-v0.12.0 (??)
+v0.11.1 (??)
 ------------------------
 
-This is a major release from 0.11.0 and includes many new features and
-enhancements along with a large number of bug fixes.
+This is a minor release from 0.11.0 and include a small number of enhances and bug fixes.
 
 API changes
 ~~~~~~~~~~~
 
 
 Enhancements
 ~~~~~~~~~~~~
-  - pd.read_html() can now parse HTML string, files or urls and return dataframes
+  - ``pd.read_html()`` can now parse HTML string, files or urls and return dataframes
     courtesy of @cpcloud. (GH3477_)
 
 See the `full release notes

doc/source/v0.12.0.txt

@@ -0,0 +1,35 @@
+.. _whatsnew_0120:
+
+v0.12.0 (??????)
+----------------
+
+This is a major release from 0.11.1 and includes many new features and
+enhancements along with a large number of bug fixes. There are also a 
+number of important API changes that long-time pandas users should
+pay close attention to.
+
+Enhancements
+~~~~~~~~~~~~
+
+- ``pd.read_msgpack()`` and ``pd.to_msgpack()`` are now a supported method of serialization
+    of arbitrary pandas (and python objects) in a lightweight portable binary format
+
+    .. ipython:: python
+
+          df = DataFrame(np.random.rand(5,2),columns=list('AB'))
+          df.to_msgpack('foo.msg')
+          pd.read_msgpack('foo.msg')
+
+          s = Series(np.random.rand(5),index=date_range('20130101',periods=5))
+          pd.to_msgpack('foo.msg', df, s)
+          pd.read_msgpack('foo.msg')
+
+    .. ipython:: python
+          :suppress:
+          :okexcept:
+
+          os.remove('foo.msg')
+
+See the `full release notes
+<https://github.com/pydata/pandas/blob/master/RELEASE.rst>`__ or issue tracker
+on GitHub for a complete list.

doc/source/whatsnew.rst

@@ -18,6 +18,8 @@ These are new features and improvements of note in each release.
 
 .. include:: v0.12.0.txt
 
+.. include:: v0.11.1.txt
+
 .. include:: v0.11.0.txt
 
 .. include:: v0.10.1.txt

pandas/io/packers.py

@@ -68,26 +68,38 @@
 except:
     _USE_MSGPACK = False
 
-def to_msgpack(path, obj, **kwargs):
+def to_msgpack(path, *args, **kwargs):
     """
     msgpack (serialize) object to input file path
 
     Parameters
     ----------
     path : string
         File path
-    obj : any object
+    args : an object or objects to serialize
+
+    append : boolean whether to append to an existing msgpack
+             (default is False)
     """
     if not _USE_MSGPACK:
         raise Exception("please install msgpack to create msgpack stores!")
-    f = open(path, 'wb')
+
+    append = kwargs.get('append')
+    if append:
+        f = open(path, 'a+b')
+    else:
+        f = open(path, 'wb')
     try:
-        f.write(msgpack.packb(obj))
+        if len(args) == 1:
+            f.write(pack(args[0]))
+        else:
+            for a in args:
+                f.write(pack(a))
     finally:
         f.close()
 
 
-def read_msgpack(path):
+def read_msgpack(path, iterator=False, **kwargs):
     """
     Load msgpack pandas object from the specified
     file path
@@ -96,15 +108,24 @@ def read_msgpack(path):
     ----------
     path : string
         File path
+    iterator : boolean, if True, return an iterator to the unpacker
+               (default is False)
 
     Returns
     -------
     obj : type of object stored in file
+
     """
     if not _USE_MSGPACK:
         raise Exception("please install msgpack to read msgpack stores!")
+    if iterator:
+        return Iterator(path)
+
     with open(path,'rb') as fh:
-        return msgpack.unpackb(fh.read())
+        l = list(unpack(fh))
+        if len(l) == 1:
+            return l[0]
+        return l
 
 dtype_dict = { 'datetime64[ns]'  : np.dtype('M8[ns]'),
                'timedelta64[ns]' : np.dtype('m8[ns]') }
@@ -296,48 +317,29 @@ def create_block(b):
         import pdb; pdb.set_trace()
         return obj
 
-def pack(o, stream, default=encode, 
-         encoding='utf-8', unicode_errors='strict'):
-    """
-    Pack an object and write it to a stream.
-    """
-
-    _packer.pack(o, stream, default=default, 
-                  encoding=encoding,
-                  unicode_errors=unicode_errors)
-def packb(o, default=encode, 
-          encoding='utf-8', unicode_errors='strict', use_single_float=False):
+def pack(o, default=encode, 
+         encoding='utf-8', unicode_errors='strict', use_single_float=False):
     """
     Pack an object and return the packed bytes.
     """
 
-    return _packer.packb(o, default=default, encoding=encoding,
-                          unicode_errors=unicode_errors, 
-                          use_single_float=use_single_float)
-
-def unpack(stream, object_hook=decode, list_hook=None, use_list=None,
-           encoding='utf-8', unicode_errors='strict', object_pairs_hook=None):
-    """
-    Unpack a packed object from a stream.
-    """
+    return Packer(default=default, encoding=encoding,
+           unicode_errors=unicode_errors, 
+           use_single_float=use_single_float).pack(o)
 
-    return _unpacker.unpack(stream, object_hook=object_hook,
-                           list_hook=list_hook, use_list=use_list, 
-                           encoding=encoding,
-                           unicode_errors=unicode_errors,
-                           object_pairs_hook=object_pairs_hook)
-def unpackb(packed, object_hook=decode, 
-            list_hook=None, use_list=None, encoding='utf-8',
-            unicode_errors='strict', object_pairs_hook=None):
+def unpack(packed, object_hook=decode, 
+           list_hook=None, use_list=False, encoding='utf-8',
+           unicode_errors='strict', object_pairs_hook=None):
     """
-    Unpack a packed object.
+    Unpack a packed object, return an iterator
+    Note: packed lists will be returned as tuples
     """
 
-    return _unpacker.unpackb(packed, object_hook=object_hook,
-                            list_hook=list_hook, 
-                            use_list=use_list, encoding=encoding,
-                            unicode_errors=unicode_errors, 
-                            object_pairs_hook=object_pairs_hook)
+    return Unpacker(packed, object_hook=object_hook,
+                    list_hook=list_hook, 
+                    use_list=use_list, encoding=encoding,
+                    unicode_errors=unicode_errors, 
+                    object_pairs_hook=object_pairs_hook)
 
 if _USE_MSGPACK:
 
@@ -352,7 +354,7 @@ def __init__(self, default=encode,
                                          use_single_float=use_single_float)
 
     class Unpacker(_unpacker.Unpacker):
-        def __init__(self, file_like=None, read_size=0, use_list=None,
+        def __init__(self, file_like=None, read_size=0, use_list=False,
                      object_hook=decode,
                      object_pairs_hook=None, list_hook=None, encoding='utf-8',
                      unicode_errors='strict', max_buffer_size=0):
@@ -365,14 +367,21 @@ def __init__(self, file_like=None, read_size=0, use_list=None,
                                            encoding=encoding, 
                                            unicode_errors=unicode_errors, 
                                            max_buffer_size=max_buffer_size)
-            
-    setattr(msgpack, 'Packer', Packer)
-    setattr(msgpack, 'Unpacker', Unpacker)
-    setattr(msgpack, 'load', unpack)
-    setattr(msgpack, 'loads', unpackb)
-    setattr(msgpack, 'dump', pack)
-    setattr(msgpack, 'dumps', packb)
-    setattr(msgpack, 'pack', pack)
-    setattr(msgpack, 'packb', packb)
-    setattr(msgpack, 'unpack', unpack)
-    setattr(msgpack, 'unpackb', unpackb)
+
+class Iterator(object):
+    """ manage the unpacking iteration,
+        close the file on completion """
+
+    def __init__(self, path, **kwargs):
+        self.path = path
+        self.kwargs = kwargs
+
+    def __iter__(self):
+
+        try:
+            fh   = open(self.path,'rb')
+            unpacker = unpack(fh)
+            for o in unpacker:
+                yield o
+        finally:
+            fh.close()