BUG: Fix wrong reading sparse matrix

[m7142yosuke]

• closes Pandas wrongly read Scipy sparse matrix #29814
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[dwhswenson]

Previously the comment in #29814 had suggested using data.indices instead of data.nonzero(). However, this PR instead strips any explicit zeros in the sparse input.

One of my use cases involves taking the difference of two sparse matrices, and then passing the result to scipy.sparse and pandas (ideally filling missing with NaN, instead of 0). There's a meaningful difference between "this element has no information from either matrix" (NaN) and "this element had the exact same value in both matrices" (0). It looks to me like this PR would obscure that distinction.

This would also make it hard to round-trip from scipy.sparse, which seems to handle explicit zeros.

Is there a solution that doesn't involve losing the information about explicit zeros?

(This wouldn't be a show-stopper for us; right now export to pandas is basically a convenience for users who want it. We'd just have to add a warning on that function.)

[WillAyd]

@jorisvandenbossche or @TomAugspurger any thoughts here?

[TomAugspurger]

What’s the performance cost of your PR be using indicies? It seems like stripping explicit zeros will require scanning the full data?

[m7142yosuke]

@TomAugspurger
Eliminating explicit zeros will require scanning the full data in my PR.
I didn't come up with other way(not scanning the full data). Is there another good way?

[jorisvandenbossche]

@m7142yosuke as I also mentioned on the issue, there has been a PR that refactored the from_spmatrix dataframe accessor method: #32825

This seems to have fixed the DataFrame bug (it has its own logic now, and does not use SparseArray.from_spmatrix under the hood), but so the SparseArray is still buggy (what you are fixing here).
But so, it's probably worth to take a look at how that PR changed the implementation, and do something similar here (or try to see if some of the code can be shared).

Also adding a test case for the original case from #29814 would be welcome

[jreback]

can you update to the comments on the PR, e.g. @jorisvandenbossche mentioned an explicit test & add a whatsnew note in 1.1., sparse bug fix section. pls merge master and ping on green.

[m7142yosuke]

@jorisvandenbossche @jreback
Please review this PR

Done.

• Refactoring code referencing PERF: optimize DataFrame.sparse.from_spmatrix performance #32825
• Add a test case for the original case from Pandas wrongly read Scipy sparse matrix #29814

[jreback]

thanks @m7142yosuke