Improve handling of numpy integers

[prashanthellina]

Numpy integers operate almost exactly like Python integers in all circumstances, but msgpack doesn't handle them. That's fine, but the error message that gets thrown is fairly confusing.

>>> import numpy
>>> five = numpy.int32(5)
>>> five
5
>>> type(five)
<type 'numpy.int32'>
>>> msgpack.dumps(five)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "_msgpack.pyx", line 151, in msgpack._msgpack.packb (msgpack/_msgpack.c:1607)
  File "_msgpack.pyx", line 135, in msgpack._msgpack.Packer.pack (msgpack/_msgpack.c:1363)
  File "_msgpack.pyx", line 130, in msgpack._msgpack.Packer._pack (msgpack/_msgpack.c:1305)
TypeError: can't serialize 5

It's not really msgpack's problem per se, but I think handling it more gracefully would be nice, since numpy is in very widespread use. I'd suggest adding type information to the TypeError, so the error message would read something like:

TypeError: can't serialize 5 (<type 'numpy.int32'>)

That's make debugging this and other similar problems easier.

Actually treating numpy integers (and other numpy scalar types) as integers would be a bonus.

Please note:
I wasn't the initial reporter of this bug/request. I am simple re-posting this (msgpack/msgpack#36 (comment)) into this repo.

[jnothman]

That's strange, because msgpack uses PyInt_Check ("Return true if o is of type PyInt_Type or a subtype of PyInt_Type.") and numpy.int32.__bases__ == (<type 'numpy.signedinteger'>, <type 'int'>) on my system (in Py2.7) and isinstance(numpy.int32(5), int) == True. The same is true of numpy's long types (and other sizes of int).

Its unsigned integer types do not inherit from Python types, but -- together with its signed and long types -- inherit from numpy.integer.

Numpy's float64 packs fine, but other float sizes do not.

Since these are all underlyingly stored as C types, perhaps it's more sensible to just special-case a check for a numpy number and do the packing in C?

The same error occurs for bools:

>>> msgpack.packb(numpy.array([True])[0])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "_packer.pyx", line 259, in msgpack._packer.packb (msgpack/_packer.cpp:259)
  File "_packer.pyx", line 184, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:184)
  File "_packer.pyx", line 179, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:179)
TypeError: can't serialize True

Here msgpack-python uses isinstance directly, but it seems numpy.bool_ is not a subtype of bool.

Strings and dicts also seem to pack fine.

[isolver]

This is a more generic issue with msppack not handling numpy data types, is it not? For example I get this issue for numpy.float32's and other dtypes as well.

Anyone working on a fix for this? If not maybe I can.

[methane]

msgpack-python now have ext type.

[dragoljub]

So is the BKM to convert to native python types? Its there a reason that np.float64 and np.int32 serialize just fine while np.float32 and np.int64? 😕

[methane]

There are no intention.
np.float64 can be serialized by accident. It may be desirialized to Python's float.