PERF: Severe performance hit on DataFrame.sum with multi index and 'skipna=False' #37976
Closed
2 of 3 tasks
Labels
Duplicate Report
Duplicate issue or pull request
Groupby
Performance
Memory or execution speed performance
Comments
miccoli
added
Bug
Needs Triage
miccoli
changed the title
|
A side note on performance. After reading #34773, I checked that timing is not influenced by In fact this is the result of For reference here the output of |
|
@miccoli thanks for your analysis. You are correct that in the case of |
jorisvandenbossche
added
Performance
|
@jorisvandenbossche I see... it was hard to catch that old issue! Basically this means that
while
So, if I got it right, introducing some NaN's in my tests I have: In [16]: pos = (123,4,5)
...: df.loc[pos] = np.NaN
...: assert np.isnan(x[pos]).all()
In [17]: s1 = df.sum(level=["i0", "i1"], skipna=False)
...: s2 = df.sum(level=["i0", "i1"], skipna=True)
In [18]: s1_ref = np.sum(x, axis=2)
...: s2_ref = np.nansum(x, axis=2)
In [19]: np.testing.assert_equal(s1.to_numpy().reshape(s1_ref.shape), s1_ref)
In [20]: np.testing.assert_array_almost_equal_nulp(
...: s2.to_numpy().reshape(s2_ref.shape), s2_ref, nulp=2
...: )Now can compute s2 also with . In [23]: sg2 = df.groupby(level=["i0", "i1"]).sum()
In [24]: assert s2.equals(sg2)
In [25]: %timeit df.groupby(level=["i0", "i1"]).sum()
59.2 ms ± 248 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)but there is no way to compute s1 ( I have a final comment/question: this implies that the preferred way forward would be to implement the |
Indeed. Only with the detail that the Lines 10518 to 10523 in 7077a08 So that is also how you can do the
Indeed (all DataFrame/Series reductions with a |
|
BTW, there is an old stalled PR for this (#15772), contributions to implement this are always welcome! (but no pressure, it's a relatively big chunk of work) |
I would love to do it, but for now I have no spare time... Please feel free to close this issue, if you think that it overlaps to much with #15675, or to keep it open if you think that it could be helpful to others. |
No problem! (and thanks for taking the time to look into this and open an issue) Yes, going to close this as a duplicate of #15675, we have enough open issues ;) |
|
Duplicate of #15675 |
jorisvandenbossche
added
Duplicate Report
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Series.sum(..., skipna=False)andDataFrame.sum(..., skipna=False)are very slow on aMultiIndex,if compared to the same method called with
(..., skipna=True)I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
(optional) I have confirmed this bug exists on the master branch of pandas.
Code Sample, a copy-pastable example
Problem description
The computation of
s1 = df.sum(level=["i0", "i1"], skipna=False)takes forever in the sample above:Here we see a severe performance hit (8.61 s vs 0.05 s) for the very same
DataFrame.sumifskipna=False.Maybe this is related to #16788 and #37622
The performance of
df.sum(level=["i0", "i1"], skipna=True)with respect to baseline numpy is not great, but I think this is not an issue here.Results are correct, fortunately. Curiously the slow result (
skipna=False) is identical with the reference numpy sum, while the faster result (skipna=True) is within ±2 ULP, but of course this is not a concern.Almost the same results hold if we use a
DataFrameinstead of aSeriesOutput of
pd.show_versions()INSTALLED VERSIONS
commit : 67a3d42
python : 3.8.2.final.0
python-bits : 64
OS : Darwin
OS-release : 19.6.0
Version : Darwin Kernel Version 19.6.0: Thu Oct 29 22:56:45 PDT 2020; root:xnu-6153.141.2.2~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.UTF-8
pandas : 1.1.4
numpy : 1.19.4
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.4
setuptools : 49.6.0
Cython : None
pytest : 6.0.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.17.0
pandas_datareader: None
bs4 : None
bottleneck : 1.3.2
fsspec : 0.8.0
fastparquet : None
gcsfs : None
matplotlib : 3.3.2
numexpr : 2.7.1
odfpy : None
openpyxl : 3.0.3
pandas_gbq : None
pyarrow : 1.0.1
pytables : None
pyxlsb : None
s3fs : 0.5.0
scipy : 1.5.3
sqlalchemy : None
tables : 3.6.1
tabulate : 0.8.7
xarray : 0.16.0
xlrd : 1.2.0
xlwt : None
numba : 0.51.1
The text was updated successfully, but these errors were encountered: