pd.concat with mixed sparse and dense data, kill structure #18594
Labels
Duplicate Report
Duplicate issue or pull request
Reshaping
Concat, Merge/Join, Stack/Unstack, Explode
Sparse
Sparse Data Type
Code Sample, a copy-pastable example if possible
Problem description
The above code fails, when using mixed type data on segmenting the frame.
A possible work-around is to recast the resulting sparse dataframe to a dense data frame via
c_new = pd.DataFrame(c)
At the bootom of this it seems that
pandas.concat
always uses the highest class object in the to catenate list, e.g. here the SparseDataFrame, to store teh resulting values in. This effectively works deep, below, however, the on surface interfaces are kind of blown out if the data contained is not as expected.This might be a very similar issue as #18551 (#18551)
Expected Output
well, hard to say if SparseDataFrames are to handle mixed type data (possibly keeping some of the columns sparse, other dense), or if
pd.concat
should fall back to a dense DataFrame here. Anyhow, the issue should be fixed one way or another,Output of
pd.show_versions()
INSTALLED VERSIONS
commit: None
python: 3.6.1.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None
pandas: 0.21.0
pytest: 3.0.7
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.13.3
scipy: 0.19.1
pyarrow: None
xarray: None
IPython: 5.3.0
sphinx: 1.5.6
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.2.2
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: 2.4.7
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.7.3
bs4: 4.6.0
html5lib: 0.999
sqlalchemy: 1.1.9
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
The text was updated successfully, but these errors were encountered: