Pandas wrongly read Scipy sparse matrix

[m7142yosuke]

Code Sample, a copy-pastable example if possible

>>> from scipy.sparse import coo_matrix
>>> sparse_data = coo_matrix((
...     [0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
...     ([0, 1, 1, 2, 2, 3, 3], [0, 1, 2, 0, 2, 0, 1])
... ))
>>> sparse_data.todense()
matrix([[0., 0., 0.],
        [0., 1., 1.],
        [1., 0., 1.],
        [1., 1., 0.]])
>>> pd.DataFrame.sparse.from_spmatrix(sparse_data)
     0    1    2
0  0.0  0.0  0.0
1  0.0  1.0  1.0
2  0.0  0.0  1.0
3  1.0  1.0  0.0
>>> pd.DataFrame.sparse.from_spmatrix(sparse_data.tocsr())
     0    1    2
0  0.0  0.0  0.0
1  0.0  1.0  1.0
2  0.0  0.0  1.0
3  1.0  1.0  0.0

Problem description

sparse_data.todense() and the other matrixes should be same, but not.
(i.e. the [2, 0] value in bottom 2 matrixes should be 1.)

Expected Output

When converting sparse matrix to Pandas DataFrame values of the sparse array should stay the same.

Output of pd.show_versions()

python : 3.7.5.final.0
pandas : 0.25.3
numpy : 1.17.4
pytz : 2019.3
dateutil : 2.8.1
pip : 19.3.1
setuptools : 41.6.0
Cython : None
pytest : 5.3.0
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.1.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : 1.3.2
sqlalchemy : None
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None

[jorisvandenbossche]

Thanks for the report! I can confirm this bug on latest master.

Under the hood, the first column of the dataframe is created as:

In [25]: a = pd.SparseArray.from_spmatrix(sparse_data.tocsc()[:, 0])   

In [26]: a 
Out[26]: 
[0.0, 0.0, 0.0, 1.0]
Fill: 0.0
IntIndex
Indices: array([2, 3], dtype=int32)

So you see that also here the result is wrong. But, the actual indices are correct (indicating [2, 3] for elements 3 and 4 not being sparse), but it's the value of the third element that is wrong (0 instead of 1). If you change the "fill_value", this becomes clearer:

In [27]: a.fill_value = np.nan

In [28]: a    
Out[28]: 
[nan, nan, 0.0, 1.0]
Fill: nan
IntIndex
Indices: array([2, 3], dtype=int32)

This might help to find the cause. Always welcome to look into it!

[m7142yosuke]

Maybe, I found the cause. This problem occurred due to unexpected sparse matrix.
Related code is below

pandas/pandas/core/arrays/sparse/array.py

Lines 418 to 422 in 79e1fc3

	arr = data.data
	idx, _ = data.nonzero()
	loc = np.argsort(idx)
	arr = arr.take(loc)
	idx.sort()

The above code is sorting sparse index in ascending order.
But the code has a bag which is assuming data.data has no explicit zero.
So, I think it is appropriate to use eliminate_zeros method if Sparse matrix has explicit zeros.

[dwhswenson]

One of my projects also got hit by this problem. I can confirm that @m7142yosuke's solution works for me. I implemented that solution as a monkey patch, and our tests pass again. dwhswenson/contact_map#69

@m7142yosuke, do you want to make a PR of this? If you don't want to, I can start a PR. But you found the bug and found the solution; you should get the credit.

It looks like anyone creating sparse dataframes from scipy.sparse matrices may be getting incorrect results from pandas, so this is an important bug to fix.

[m7142yosuke]

@dwhswenson
Thank you for your comment.
I thought my solution was wrong, because no reply.
I want to make a PR

[jorisvandenbossche]

So on latest master, the original bug report now is correct:

In [3]: pd.DataFrame.sparse.from_spmatrix(sparse_data) 
Out[3]: 
     0    1    2
0  0.0  0.0  0.0
1  0.0  1.0  1.0
2  1.0  0.0  1.0
3  1.0  1.0  0.0

This might be due to #32825, a PR that refactored from_spmatrix for performance.

But, the SparseArray.from_spmatrix is still buggy:

In [7]: pd.arrays.SparseArray.from_spmatrix(sparse_data.tocsc()[:, 0]) 
Out[7]: 
[0.0, 0.0, 0.0, 1.0]
Fill: 0.0
IntIndex
Indices: array([2, 3], dtype=int32)

[HYChou0515]

Does #28992 related?