BUG: expanding numba functions being cached with arguments

[lucaswebb]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample

def expanding_multiply(values, x):
    arr = values[-1]
    arr[1] = values[:,0].sum() * x
    return arr

df = pd.DataFrame({'id': [1, 2, 3], 'value': [0, 0, 0]})

def call_expanding(x):
    expanded = df.expanding(method="table").apply(expanding_multiply, raw=True, engine="numba", args=x)
    return expanded

call_expanding(tuple([1]))
#  1.0    1.0
#  2.0    3.0
#  3.0    6.0

call_expanding(tuple([2]))
#  1.0    1.0
#  2.0    3.0
#  3.0    6.0

Problem description

When using expanding.apply with engine='numba' calling the same function with different arguments gives the same results as the first time it was called. I believe this is because the function is being cached with its arguments.

pandas/pandas/core/window/rolling.py

Lines 1172 to 1175 in e8dbdb0

	apply_func = generate_numba_table_func(
	args, kwargs, func, engine_kwargs, f"{caller_name}_apply"
	)
	numba_cache_key = (func, f"{caller_name}_apply_table")

Expected Output

call_expanding(tuple([2]))
# 1.0    2.0
# 2.0    6.0
# 3.0    12.0

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 2dd9e9b
python : 3.7.7.final.0
python-bits : 64
OS : Darwin
OS-release : 19.5.0
Version : Darwin Kernel Version 19.5.0: Thu Apr 30 18:25:59 PDT 2020; root:xnu-6153.121.1~7/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : en_US.UTF-8
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.3.0rc1
numpy : 1.19.1
pytz : 2019.1
dateutil : 2.8.1
pip : 19.2.3
setuptools : 41.2.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.3
html5lib : None
pymysql : None
psycopg2 : 2.8.2 (dt dec pq3 ext lo64)
jinja2 : 3.0.1
IPython : 7.18.1
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.6.3
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : 0.53.1

[lucaswebb]

This problem affects all windows.rolling and windows.expanding .apply functions. One solution is to pass *args in to the apply functions instead of args: tuple.

pandas/pandas/core/window/rolling.py

Lines 1144 to 1152 in e8dbdb0

	def apply(
	self,
	func: Callable[..., Any],
	raw: bool = False,
	engine: str | None = None,
	engine_kwargs: dict[str, bool] | None = None,
	args: tuple[Any, ...] | None = None,
	kwargs: dict[str, Any] | None = None,
	):

pandas/pandas/core/window/rolling.py

Line 525 in e8dbdb0

return func(x, start, end, min_periods)

becomes

return func(x, start, end, min_periods, args)

[mzeitlin11]

Thanks for reporting and investigating this @lucaswebb! This looks similar to #41647, would you be interested in putting up a pr to fix this? cc @mroeschke