groupby().transform(f) is very slow if there are assignment statements to the argument of f()

[ruoyu0088]

Here is the test code:

import pandas as pd
from numpy.random import randint, rand, seed

seed(10)
N_Group = 20
N_Rows = 1000
df = pd.DataFrame({"g":randint(0, N_Group, N_Rows), "a":rand(N_Rows), "b":rand(N_Rows)})

def f(df):
    df["a"] -= df["b"].mean()
    return df

%time df1 = df.groupby("g").apply(f)
%time df2 = df.groupby("g").transform(f)

the output is:

Wall time: 17 ms
Wall time: 682 ms

before the for-loop for every group in apply(), it disable the 'mode.chained_assignment' opition:

    # ignore SettingWithCopy here in case the user mutates
    with option_context('mode.chained_assignment',None):
        return self._python_apply_general(f)

but there is not such code for transform().

[shoyer]

This may technically work, but it's really not a good idea to use in-place operations in a groupby operation. For example, see the warning here. So I have mixed feelings about optimizing this.

[jreback]

I'll put it on the list. A PR would be helpful here. I agree with @shoyer here. IMHO modifying things in a groupby should simply be banned. This is at odds with pure functions and all semantics except where explict in pandas

[rhshadrach]

Duplicate of #12653