pd.Series.equals() fails for Series containing iterable objects

[bfollinprm]

Code Sample, a copy-pastable example if possible

pd.Series(
    [np.array([1, 2]), np.array([1, 2])]
).equals(pd.Series(
    [np.array([1, 2]), np.array([1, 2])]
))

# Evaluates to False

Problem description

For objects that overload equality with elementwise equality (this includes np.array and pd.Series objects), pd.Series.equals() catches

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

and returns False, behavior inherited from np.equal applied to object arrays. While you can argue this is a numpy bug, in numpy equality is purposefully strict for dtype=='object'. In pandas, it is a common enough procedure to tabulate arrays and lists under a single column (which may contain a named (time)series, image array, or an unpacked NLP document), and this behavior of equality is unintuitive.

Expected Output

True

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 2.7.13.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-1048-aws
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.22.0
pytest: 3.2.2
pip: 9.0.1
setuptools: 27.2.0
Cython: None
numpy: 1.13.3
scipy: 0.19.1
pyarrow: None
xarray: None
IPython: 5.3.0
sphinx: 1.5.4
patsy: None
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.0
html5lib: 0.9999999
sqlalchemy: 1.1.9
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[jorisvandenbossche]

Storing lists or arrays in Series/DataFrame is generally not fully supported (in the sense that: it works for basic things, but may fail for corner cases (eg indexing, setting values, ..) and it's not well tested, and it's not something the core devs will typically give priority to).

But if you have an relative straightforward fix, we will be happy to take a PR.

Note that numpy also does not handle equality well for object arrays of arrays:

In [62]: s = pd.Series([np.array([1, 2]), np.array([1, 2])])

In [63]: s.values
Out[63]: array([array([1, 2]), array([1, 2])], dtype=object)

In [64]: s.values == s.values
/home/joris/miniconda3/envs/dev/bin/ipython:1: DeprecationWarning: elementwise == comparison failed; this will raise an error in the future.
  #!/home/joris/miniconda3/envs/dev/bin/python
Out[64]: True

[jorisvandenbossche]

Just a thought: it might be that exploring an ExtensionArray (new in master: http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#extending-pandas-with-custom-types) might be a way to fix this.

[jreback]

yeah this is really a numpy issue as that is how we compare object arrays. If we need to peer inside those arrays we have to do an inference check, which could be done here. Storing list-likes inside a pandas cell is not supported at all and causes lots of pain in our codebase (think indexing). Using an extension array would be a real nice usecase here.

[mroeschke]

This looks to work on master. I guess it could use a test

In [50]: pd.Series(
    ...:     [np.array([1, 2]), np.array([1, 2])]
    ...: ).equals(pd.Series(
    ...:     [np.array([1, 2]), np.array([1, 2])]
    ...: ))
Out[50]: True

In [51]: pd.__version__
Out[51]: '1.1.0.dev0+1216.gd4d58f960'

[avinashpancham]

take