np.float16 support

[datapythonista]

open issues & xref

• (fillna) fillna can't process np.float16 type #19205
• Incorrect usage of float16 in algos.pyx #10382
• BUG: DataFrame.reindex sometimes promotes float16 / float32 #15517
• read_csv() option to parse numeric columns to np.float32 or np.float16 #2511
• pandas.io.json.dumps raises OverflowError on float32 and float16 NaNs #16686

think this should be a bug. When calling the fill_na method in a pandas Series of type np.float16 it raises a ValueError, because of invalid dtype. May be there is a reason for that I can't guess, but I think it's a bug because it does not detect np.float16 as a subclass of np.floating. I'm using Python 3.4, didn't test if it also fails in Python 2. It fails for both methods, 'pad' and 'bfill'.

For np.float32 and np.float64 it works correctly. This code reproduces the error with np.float16:

import numpy as np
import pandas as pd

s = pd.Series(np.random.randn(100), dtype=np.float16)
rets = s.fillna(method='bfill')

and this is the ValueError it generates:

---

ValueError                                Traceback (most recent call last)
<ipython-input-133-5610749057ca> in <module>()
      3 
      4 s = pd.Series(np.random.randn(100), dtype=np.float16)
----> 5 rets = s.fillna(method='bfill')

/opt/anaconda3/lib/python3.4/site-packages/pandas/core/generic.py in fillna(self, value, method, axis, inplace, limit, downcast)
   2227                                               inplace=inplace,
   2228                                               coerce=True,
-> 2229                                               downcast=downcast)
   2230         else:
   2231             if method is not None:

/opt/anaconda3/lib/python3.4/site-packages/pandas/core/internals.py in interpolate(self, **kwargs)
   2223 
   2224     def interpolate(self, **kwargs):
-> 2225         return self.apply('interpolate', **kwargs)
   2226 
   2227     def shift(self, **kwargs):

/opt/anaconda3/lib/python3.4/site-packages/pandas/core/internals.py in apply(self, f, axes, filter, do_integrity_check, **kwargs)
   2190                                                  copy=align_copy)
   2191 
-> 2192             applied = getattr(b, f)(**kwargs)
   2193 
   2194             if isinstance(applied, list):

/opt/anaconda3/lib/python3.4/site-packages/pandas/core/internals.py in interpolate(self, method, axis, index, values, inplace, limit, fill_value, coerce, downcast, **kwargs)
    648                                                fill_value=fill_value,
    649                                                coerce=coerce,
--> 650                                                downcast=downcast)
    651         # try an interp method
    652         try:

/opt/anaconda3/lib/python3.4/site-packages/pandas/core/internals.py in _interpolate_with_fill(self, method, axis, inplace, limit, fill_value, coerce, downcast)
    693                                     limit=limit,
    694                                     fill_value=fill_value,
--> 695                                     dtype=self.dtype)
    696         values = self._try_coerce_result(values)
    697 

/opt/anaconda3/lib/python3.4/site-packages/pandas/core/common.py in interpolate_2d(values, method, axis, limit, fill_value, dtype)
   1576         values = transf(pad_2d(transf(values), limit=limit, mask=mask, dtype=dtype))
   1577     else:
-> 1578         values = transf(backfill_2d(transf(values), limit=limit, mask=mask, dtype=dtype))
   1579 
   1580     # reshape back

/opt/anaconda3/lib/python3.4/site-packages/pandas/core/common.py in backfill_2d(values, limit, mask, dtype)
   1378 
   1379     if _method is None:
-> 1380         raise ValueError('Invalid dtype for backfill_2d [%s]' % dtype.name)
   1381 
   1382     if mask is None:

ValueError: Invalid dtype for backfill_2d [float16]

And my guess is that the problem is this part of the code, I can debug it once you confirm this is a bug:
https://github.com/pydata/pandas/blob/master/pandas/core/common.py#L2459

[jreback]

this is not implemented. Generally float16 are not available on all platforms so pandas does not support these (I did try to make it work, but was a rabbit hole). Welcome to give it a go if you'd like.

[jreback]

http://en.wikipedia.org/wiki/Half-precision_floating-point_format

generally since operations between lower precision dtypes will silently fail using lower precision is generally not a great idea.

[datapythonista]

Sorry to change the title, didn't see that you change it so quick, and I thought it was some kind of github autosuggestion that I didn't see before. ;)

[jreback]

np

happy to have this implemted
I just remember their were some issues but could have been something else

[datapythonista]

In case someone else has the same problem, my actual code error was on the pct_change() method, and it can be fixed, assuming that the Series has no missing values, with the attribute method set to None as in this sample:

import numpy as np
import pandas as pd

s = pd.Series(np.random.randn(100), dtype=np.float16)
rets = s.pct_change(method='bfill')

Anyway, I'll give it a try, and see if I can add support for the np.float16 dtype to the fill_na method.

[gustavz]

any news on this? I still see errors when using float16.
In my case I get TypeError: No matching signature found on running apply and fillna methods on float16

[jreback]

happy to take pull requests but we have very little float16 support / and very little testing

[mw66]

Wow, 5 years now, still an open issue.

[jreback]

Wow, 5 years now, still an open issue.

there are about 3400 open issues and a few volunteer core devs

it’s all about community contributions / or lack thereof

[torfsen]

While I can understand that complete support for float16 is currently not a priority, it would be nice to document the limited support for float16 in the expected places. In particular, the examples in https://pandas.pydata.org/docs/user_guide/basics.html#basics-dtypes do show float16 being used, and do not mention any limitations anywhere.

I stumbled upon this issue when I tried to reindex a DataFrame with float16 values. Quick demo (pandas 1.4.1, numpy 1.22.3):

In [44]: df = pd.DataFrame({"x": [10, 20, 30], "y": [40, 50, 60], "z": [70, 80, 90]}, index=[1, 2, 3], dtype="float32")

In [45]: df
Out[45]: 
      x     y     z
1  10.0  40.0  70.0
2  20.0  50.0  80.0
3  30.0  60.0  90.0

In [46]: df.reindex(index=[0, 1, 2], columns=["y", "z"])
Out[46]: 
      y     z
0   NaN   NaN
1  40.0  70.0
2  50.0  80.0

In [47]: df.astype("float16").reindex(index=[0, 1, 2], columns=["y", "z"])
Out[47]: 
      y     z
0  60.0  90.0
1  40.0  70.0
2  50.0  80.0

(The reindexing result for float16 contains some arbitrary values instead of the expected NaNs for the missing labels)

[mroeschke]

We specifically raise for np.float16 now in constructors so going to close