Skip to content

concat([sparse]) should be a Series / DataFrame #25702

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
TomAugspurger opened this issue Mar 13, 2019 · 3 comments · Fixed by #25719
Closed

concat([sparse]) should be a Series / DataFrame #25702

TomAugspurger opened this issue Mar 13, 2019 · 3 comments · Fixed by #25719
Labels
Sparse Sparse Data Type
Milestone
0.25.0

Comments

@TomAugspurger
Copy link
Contributor

I messed up the logic somewhere along the way.

In [11]: df = pd.DataFrame({"A": pd.SparseArray([0, 1])})

In [12]: type(pd.concat([df, df]))
Out[12]: pandas.core.sparse.frame.SparseDataFrame

That should be a DataFrame with sparse values.

When should concat return a SparseSeries/Frame?

a.) When any of the inputs are SparseSeries / SparseDataFrame
b.) When all of the inputs are SparseSeries/Frame.

Option b is probably least disruptive. Implementing it isn't a pain.
@TomAugspurger TomAugspurger added the Sparse Sparse Data Type label Mar 13, 2019
@TomAugspurger TomAugspurger added this to the 0.25.0 milestone Mar 13, 2019
@WillAyd
Copy link
Member

WillAyd commented Mar 13, 2019

What's the downside to option a? I admittedly don't use these objects a lot but to cast a vote in there that seems more intuitive to me

@TomAugspurger
Copy link
Contributor Author

TomAugspurger commented Mar 13, 2019 via email

@TomAugspurger
Copy link
Contributor Author

Note that this only affects DataFrame. concat(List[Series[Sparse]]) has the desired behavior of returning a Series with sparse values.

In [4]: type(pd.concat([pd.Series(pd.SparseArray([0, 1]))] * 2))
Out[4]: pandas.core.series.Series

TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Mar 13, 2019
@TomAugspurger
API: concat on sparse values
96508ca 
API breaking change to `concat(List[DataFrame[Sparse]])` to
return a DataFrame with sparse values, rather than a SparseDataFrame.

Doing an outright break, rather than deprecation, because I have a
followup PR deprecating SparseDataFrame. We return this internally in
a few places (e.g. get_dummies on all-sparse data).

Closes pandas-dev#25702
@TomAugspurger TomAugspurger mentioned this issue Mar 13, 2019
Merged
4 tasks
TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Mar 15, 2019
@TomAugspurger
Squashed commit of the following:
2919be4 
commit 96508ca
Author: Tom Augspurger <[email protected]>
Date:   Wed Mar 13 08:23:21 2019 -0500

    API: concat on sparse values

    API breaking change to `concat(List[DataFrame[Sparse]])` to
    return a DataFrame with sparse values, rather than a SparseDataFrame.

    Doing an outright break, rather than deprecation, because I have a
    followup PR deprecating SparseDataFrame. We return this internally in
    a few places (e.g. get_dummies on all-sparse data).

    Closes pandas-dev#25702
jreback pushed a commit that referenced this issue Mar 19, 2019
@TomAugspurger @jreback
API: concat on sparse values (#25719)
8311214 
API breaking change to `concat(List[DataFrame[Sparse]])` to
return a DataFrame with sparse values, rather than a SparseDataFrame.

Doing an outright break, rather than deprecation, because I have a
followup PR deprecating SparseDataFrame. We return this internally in
a few places (e.g. get_dummies on all-sparse data).

Closes #25702
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Sparse Sparse Data Type
Projects
None yet
Milestone
0.25.0
Development

Successfully merging a pull request may close this issue.

API: concat on sparse values TomAugspurger/pandas
2 participants
@WillAyd @TomAugspurger