drop_duplicates throws error when series stores numpy array

[garmilla]

Code Sample

import pandas as pd
import numpy as np
df = pd.DataFrame({'A' : []})
df.loc[0] = [np.array(['hello'])]
df.A.drop_duplicates()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
TypeError: unhashable type: 'numpy.ndarray'

During handling of the above exception, another exception occurred:

SystemError                               Traceback (most recent call last)
<ipython-input-7-311de327a529> in <module>
----> 1 df.A.drop_duplicates()

/usr/local/lib/python3.5/dist-packages/pandas/core/series.py in drop_duplicates(self, keep, inplace)
   1739         Name: animal, dtype: object
   1740         """
-> 1741         return super(Series, self).drop_duplicates(keep=keep, inplace=inplace)
   1742 
   1743     def duplicated(self, keep='first'):

/usr/local/lib/python3.5/dist-packages/pandas/core/base.py in drop_duplicates(self, keep, inplace)
   1507                 return self._shallow_copy()
   1508 
-> 1509         duplicated = self.duplicated(keep=keep)
   1510         result = self[np.logical_not(duplicated)]
   1511         if inplace:

/usr/local/lib/python3.5/dist-packages/pandas/core/series.py in duplicated(self, keep)
   1813         dtype: bool
   1814         """
-> 1815         return super(Series, self).duplicated(keep=keep)
   1816 
   1817     def idxmin(self, axis=0, skipna=True, *args, **kwargs):

/usr/local/lib/python3.5/dist-packages/pandas/core/base.py in duplicated(self, keep)
   1521             return duplicated(self, keep=keep)
   1522         else:
-> 1523             return self._constructor(duplicated(self, keep=keep),
   1524                                      index=self.index).__finalize__(self)
   1525 

/usr/local/lib/python3.5/dist-packages/pandas/core/algorithms.py in duplicated(values, keep)
    785     values, dtype, ndtype = _ensure_data(values)
    786     f = getattr(htable, "duplicated_{dtype}".format(dtype=ndtype))
--> 787     return f(values, keep=keep)
    788 
    789 

SystemError: <built-in function duplicated_object> returned a result with an error set

Problem description

If I add another row with the same value drop_duplicates doesn't throw an exception but fails to remove the duplicate. If I add a third row with the same value it removes only one of the three duplicates.

df.loc[1] = [np.array(['hello'])]
df.A.drop_duplicates()
0    [hello]
1    [hello]
Name: A, dtype: object
df.loc[2] = [np.array(['hello'])]
df.A.drop_duplicates()
0    [hello]
1    [hello]
Name: A, dtype: object

Expected Output

0    [hello]
Name: A, dtype: object

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.15.0-45-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.2
pytest: 3.4.2
pip: 19.0.3
setuptools: 38.5.1
Cython: None
numpy: 1.16.2
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 7.0.1
sphinx: None
patsy: 0.5.0
dateutil: 2.8.0
pytz: 2018.9
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 3.0.0
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: 0.7.3
lxml.etree: 4.1.1
bs4: 4.4.1
html5lib: 1.0b8
sqlalchemy: 1.2.4
pymysql: None
psycopg2: 2.7.4 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

[paste the output of pd.show_versions() here below this line]

[mroeschke]

pandas doesn't have first class support for storing & operating on non-scalar data. This would be appropriate for an ExtensionArray that knows how to operate on 1D data structures.

[WillAyd]

This is essentially a duplicate of #12693

[garmilla]

@mroeschke thanks for the suggestion, I didn't know about ExtensionArrays, will definitely try to use them for this task.

@WillAyd sorry, didn't see that before posting.