Potential sparse memory leak

[wesm]

from SO:

# In [4]: type(pd)
# Out[4]: pandas.sparse.frame.SparseDataFrame
memid = unique(pd.Member)
pan = {}
for mem in memid:
    pan[mem] = pd[pd.Member==mem]
goal = pandas.Panel(pan)

comments

What are the dimensions (pd.shape) and density (pd.density) of the SparseDataFrame? Any change you can e-mail me a pickle of the object (pd.save(file_path)) for me to have a look to try to diagnose what's up? BTW these questions would be better posed on the mailing list than SO. – wesm 29 mins ago 


pd.shape = (2668990, 232) – user1162837 23 mins ago


pd.density = 0.12814551216649045 The file is too large to send through email. – user1162837 22 mins ago


Also, pandas is a great library. :) – user1162837 14 mins ago

[wesm]

After some memory profiling this appears to be fine