Serialize numpy scalars in pd.msgpack

[mrocklin]

Code Sample, a copy-pastable example if possible

import numpy as np
import pandas as pd
pd.msgpack.dumps(np.int32(0))

Currently procduces:

TypeError: can't serialize 0

Expected Output

b'\x00'

output of pd.show_versions()

In [5]: pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.4.4.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-59-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.17.0rc1+752.gfe201a2
nose: 1.3.7
pip: 8.0.3
setuptools: 20.1.1
Cython: 0.22.1
numpy: 1.10.4
scipy: 0.15.1
statsmodels: 0.6.1
IPython: 4.0.1
sphinx: 1.3.1
patsy: 0.3.0
dateutil: 2.4.2
pytz: 2015.4
blosc: 1.2.8
bottleneck: 1.0.0
tables: 3.2.0
numexpr: 2.4.3
matplotlib: 1.4.3
openpyxl: 1.8.5
xlrd: 0.9.3
xlwt: 1.0.0
xlsxwriter: 0.7.3
lxml: 3.4.4
bs4: 4.3.2
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.5
pymysql: None
psycopg2: None
jinja2: 2.8

Related to msgpack/msgpack-python#61

[jreback]

hmm, I thought this should work and was tested. guess not. Should be a simple fix. Where did you encounter in the wild? I don't think you can actually do this with a pandas object (at least it won't generate a numpy scalar directly).

[mrocklin]

I ran into it originally with df.memory_usage().sum() in an older version of Pandas. This doesn't recur in master. Still, I would like to be able to depend upon the msgpack implementation in Pandas for general computations, where this might arise.

[jorisvandenbossche]

@jreback @mrocklin status of this? If it's an easy fix, maybe we can still include it in 0.19.0?

[jorisvandenbossche]

@jreback Seems that adding a

elif isinstance(o, np.object):
    o = o.item()
    continue

in _packer.Packers_pack() fixes the issue. But is that the level where you would want to fix this? (alternatively, the Packer itself could still be unaware of numpy scalars, and could convert them beforehand in the msgpack.pack method before passing the value to the Packer).

[jreback]

we have a routine to detect
zero_dim_ndarray (and extract) in pandas.lib

[samuelsinayoko]

Happy to help out with this one. Are we meant to dig into pandas/io/msgpack/_packer.pyx to get this fixed?

[jorisvandenbossche]

@samuelsinayoko the msgpack support is deprecated in pandas, so I don't think it is worth improving it.

[simonjayhawkins]

msgpack is deprecated #30112