Skip to content

RLS: 2.1.3 #55713

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
lithomas1 opened this issue Oct 26, 2023 · 11 comments
Closed

RLS: 2.1.3 #55713

lithomas1 opened this issue Oct 26, 2023 · 11 comments
Labels
Release
Milestone
2.1.3

Comments

@lithomas1
Copy link
Member

lithomas1 commented Oct 26, 2023 •
edited
Loading

Release date: 11/9

I am thinking that this should be the last release in the 2.1 series, since I want to release 2.2 (which should hopefully be compatible with numpy 2.0) before numpy does their 2.0 release.

cc @pandas-dev/pandas-core
@lithomas1 lithomas1 added this to the 2.1.3 milestone Oct 26, 2023
@jorisvandenbossche
Copy link
Member

Because of #55894, I am planning to do an earlier 2.1.3 tomorrow, if all goes well

@lithomas1
Copy link
Member Author

Because of #55894, I am planning to do an earlier 2.1.3 tomorrow, if all goes well

@pandas-dev/pandas-core Please don't merge to 2.1.x anymore, if anything didn't get in in time we'll do a 2.1.4 release.

@jorisvandenbossche
Copy link
Member

Done! -> https://github.com/pandas-dev/pandas/releases/tag/v2.1.3 and https://pypi.org/project/pandas/2.1.3

@lithomas1
Copy link
Member Author

@jorisvandenbossche
Do you want me to trigger the conda-forge builds?

Their bot usually does it automatically but it takes a while for it to do it.

@jorisvandenbossche
Copy link
Member

I was planning to wait on the bot, but feel free to trigger it manually.

@lithomas1
Copy link
Member Author

OK, I did it manually.

(For future reference, you can open an issue on pandas-feedstock with the title @conda-forge-admin, please update version to trigger it).

@jorisvandenbossche
Copy link
Member

Ah, that's easy ;) Something else to update in the release guide!

@lithomas1
Copy link
Member Author

Looks like the segfault in rolling returned :(

  File "/Users/runner/miniforge3/conda-bld/pandas_1699651050096/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.10/site-packages/pandas/core/window/rolling.py", line 1824 in corr_func
  File "/Users/runner/miniforge3/conda-bld/pandas_1699651050096/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.10/site-packages/pandas/core/window/common.py", line 20 in flex_binary_moment
  File "/Users/runner/miniforge3/conda-bld/pandas_1699651050096/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.10/site-packages/pandas/core/window/rolling.py", line 558 in _apply_pairwise
  File "/Users/runner/miniforge3/conda-bld/pandas_1699651050096/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.10/site-packages/pandas/core/window/rolling.py", line 1845 in corr
  File "/Users/runner/miniforge3/conda-bld/pandas_1699651050096/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.10/site-packages/pandas/core/window/rolling.py", line 2852 in corr
  File "/Users/runner/miniforge3/conda-bld/pandas_1699651050096/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.10/site-packages/pandas/tests/window/test_pairwise.py", line 115 in test_rolling_corr_with_zero_variance

Here's the test that segfaults

pandas/pandas/tests/window/test_pairwise.py

Lines 110 to 115 in 09ed69e

def test_rolling_corr_with_zero_variance(window):
# GH 18430
s = Series(np.zeros(20))
other = Series(np.arange(20))
assert s.rolling(window=window).corr(other=other).isna().all()

@WillAyd
Copy link
Member

WillAyd commented Nov 11, 2023

Here is the ASAN output - I assume this is an off by one in roll_mean_pandas

==92164==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60e0000048a0 at pc 0x7faabc2f8f39 bp 0x7ffc691500b0 sp 0x7ffc691500a8
READ of size 8 at 0x60e0000048a0 thread T0
    #0 0x7faabc2f8f38 in __pyx_pf_6pandas_5_libs_6window_12aggregations_2roll_mean pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so.p/pandas/_libs/window/aggregations.pyx.cpp:5258
    #1 0x7faabc2f6c89 in __pyx_pw_6pandas_5_libs_6window_12aggregations_3roll_mean pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so.p/pandas/_libs/window/aggregations.pyx.cpp:5000
    #2 0x55926ff7bc55 in cfunction_call /usr/local/src/conda/python-3.11.6/Objects/methodobject.c:542
    #3 0x55926ff5bbd2 in _PyObject_MakeTpCall /usr/local/src/conda/python-3.11.6/Objects/call.c:214
    #4 0x55926ff68462 in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:4760
    #5 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #6 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #7 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #8 0x55926ff94d9f in _PyVectorcall_Call /usr/local/src/conda/python-3.11.6/Objects/call.c:257
    #9 0x55926ff94d9f in _PyObject_Call /usr/local/src/conda/python-3.11.6/Objects/call.c:328
    #10 0x55926ff94d9f in PyObject_Call /usr/local/src/conda/python-3.11.6/Objects/call.c:355
    #11 0x55926ff6c55a in do_call_core /usr/local/src/conda/python-3.11.6/Python/ceval.c:7343
    #12 0x55926ff6c55a in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:5367
    #13 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #14 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #15 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #16 0x55926ff6c55a in do_call_core /usr/local/src/conda/python-3.11.6/Python/ceval.c:7343
    #17 0x55926ff6c55a in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:5367
    #18 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #19 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #20 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #21 0x55926ff5f40f in _PyObject_FastCallDictTstate /usr/local/src/conda/python-3.11.6/Objects/call.c:152
    #22 0x55926ff92aa8 in _PyObject_Call_Prepend /usr/local/src/conda/python-3.11.6/Objects/call.c:482
    #23 0x559270063568 in slot_tp_call /usr/local/src/conda/python-3.11.6/Objects/typeobject.c:7623
    #24 0x55926ff5bbd2 in _PyObject_MakeTpCall /usr/local/src/conda/python-3.11.6/Objects/call.c:214
    #25 0x55926ff68462 in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:4760
    #26 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #27 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #28 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #29 0x55926ff6c55a in do_call_core /usr/local/src/conda/python-3.11.6/Python/ceval.c:7343
    #30 0x55926ff6c55a in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:5367
    #31 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #32 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #33 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #34 0x55926ff5f40f in _PyObject_FastCallDictTstate /usr/local/src/conda/python-3.11.6/Objects/call.c:152
    #35 0x55926ff92aa8 in _PyObject_Call_Prepend /usr/local/src/conda/python-3.11.6/Objects/call.c:482
    #36 0x559270063568 in slot_tp_call /usr/local/src/conda/python-3.11.6/Objects/typeobject.c:7623
    #37 0x55926ff94d2d in _PyObject_Call /usr/local/src/conda/python-3.11.6/Objects/call.c:343
    #38 0x55926ff94d2d in PyObject_Call /usr/local/src/conda/python-3.11.6/Objects/call.c:355
    #39 0x55926ff6c55a in do_call_core /usr/local/src/conda/python-3.11.6/Python/ceval.c:7343
    #40 0x55926ff6c55a in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:5367
    #41 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #42 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #43 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #44 0x55926ff6c55a in do_call_core /usr/local/src/conda/python-3.11.6/Python/ceval.c:7343
    #45 0x55926ff6c55a in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:5367
    #46 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #47 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #48 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #49 0x55926ff6c55a in do_call_core /usr/local/src/conda/python-3.11.6/Python/ceval.c:7343
    #50 0x55926ff6c55a in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:5367
    #51 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #52 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #53 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #54 0x55926ff5f40f in _PyObject_FastCallDictTstate /usr/local/src/conda/python-3.11.6/Objects/call.c:152
    #55 0x55926ff92aa8 in _PyObject_Call_Prepend /usr/local/src/conda/python-3.11.6/Objects/call.c:482
    #56 0x559270063568 in slot_tp_call /usr/local/src/conda/python-3.11.6/Objects/typeobject.c:7623
    #57 0x55926ff5bbd2 in _PyObject_MakeTpCall /usr/local/src/conda/python-3.11.6/Objects/call.c:214
    #58 0x55926ff68462 in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:4760
    #59 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #60 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #61 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #62 0x55926ff6c55a in do_call_core /usr/local/src/conda/python-3.11.6/Python/ceval.c:7343
    #63 0x55926ff6c55a in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:5367
    #64 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #65 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #66 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #67 0x55926ff5f40f in _PyObject_FastCallDictTstate /usr/local/src/conda/python-3.11.6/Objects/call.c:152
    #68 0x55926ff92aa8 in _PyObject_Call_Prepend /usr/local/src/conda/python-3.11.6/Objects/call.c:482
    #69 0x559270063568 in slot_tp_call /usr/local/src/conda/python-3.11.6/Objects/typeobject.c:7623
    #70 0x55926ff5bbd2 in _PyObject_MakeTpCall /usr/local/src/conda/python-3.11.6/Objects/call.c:214
    #71 0x55926ff68462 in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:4760
    #72 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #73 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #74 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #75 0x55926ff6c55a in do_call_core /usr/local/src/conda/python-3.11.6/Python/ceval.c:7343
    #76 0x55926ff6c55a in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:5367
    #77 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #78 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #79 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #80 0x55926ff5f40f in _PyObject_FastCallDictTstate /usr/local/src/conda/python-3.11.6/Objects/call.c:152
    #81 0x55926ff92aa8 in _PyObject_Call_Prepend /usr/local/src/conda/python-3.11.6/Objects/call.c:482
    #82 0x559270063568 in slot_tp_call /usr/local/src/conda/python-3.11.6/Objects/typeobject.c:7623
    #83 0x55926ff5bbd2 in _PyObject_MakeTpCall /usr/local/src/conda/python-3.11.6/Objects/call.c:214
    #84 0x55926ff68462 in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:4760
    #85 0x55927001ecbc in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #86 0x55927001ecbc in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #87 0x55927001e34e in PyEval_EvalCode /usr/local/src/conda/python-3.11.6/Python/ceval.c:1140
    #88 0x559270035c9d in builtin_exec_impl /usr/local/src/conda/python-3.11.6/Python/bltinmodule.c:1077
    #89 0x559270035c9d in builtin_exec /usr/local/src/conda/python-3.11.6/Python/clinic/bltinmodule.c.h:465
    #90 0x55926ff750ee in cfunction_vectorcall_FASTCALL_KEYWORDS /usr/local/src/conda/python-3.11.6/Objects/methodobject.c:443
    #91 0x55926ff74fdb in _PyObject_VectorcallTstate /usr/local/src/conda/python-3.11.6/Include/internal/pycore_call.h:92
    #92 0x55926ff74fdb in PyObject_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:299
    #93 0x55926ff68462 in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:4760
    #94 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #95 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #96 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #97 0x559270048097 in pymain_run_module /usr/local/src/conda/python-3.11.6/Modules/main.c:300
    #98 0x559270047a08 in pymain_run_python /usr/local/src/conda/python-3.11.6/Modules/main.c:595
    #99 0x559270047a08 in Py_RunMain /usr/local/src/conda/python-3.11.6/Modules/main.c:680
    #100 0x55927000d006 in Py_BytesMain /usr/local/src/conda/python-3.11.6/Modules/main.c:734
    #101 0x7fab336280cf in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #102 0x7fab33628188 in __libc_start_main_impl ../csu/libc-start.c:360
    #103 0x55927000ceac  (/home/willayd/mambaforge/envs/pandas-dev/bin/python3.11+0x28feac)

0x60e0000048a0 is located 0 bytes to the right of 160-byte region [0x60e000004800,0x60e0000048a0)
allocated by thread T0 here:
    #0 0x7fab33a955f8 in __interceptor_malloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x7fab2f9105ce in PyDataMem_UserNEW (/home/willayd/mambaforge/envs/pandas-dev/lib/python3.11/site-packages/numpy/core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so+0x1105ce)

SUMMARY: AddressSanitizer: heap-buffer-overflow pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so.p/pandas/_libs/window/aggregations.pyx.cpp:5258 in __pyx_pf_6pandas_5_libs_6window_12aggregations_2roll_mean
Shadow bytes around the buggy address:
  0x0c1c7fff88c0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c1c7fff88d0: 00 00 00 00 00 00 00 00 00 00 00 00 fa fa fa fa
  0x0c1c7fff88e0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1c7fff88f0: 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa fa
  0x0c1c7fff8900: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c1c7fff8910: 00 00 00 00[fa]fa fa fa fa fa fa fa 00 00 00 00
  0x0c1c7fff8920: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1c7fff8930: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c1c7fff8940: 00 00 00 00 00 00 00 00 00 00 00 00 fa fa fa fa
  0x0c1c7fff8950: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1c7fff8960: 00 00 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==92164==ABORTING

@WillAyd
Copy link
Member

WillAyd commented Nov 11, 2023

The offending line in Cython is this:

            /* "pandas/_libs/window/aggregations.pyx":286
 *                 compensation_add = compensation_remove = sum_x = 0
 *                 nobs = neg_ct = 0
 *                 prev_value = values[s]             # <<<<<<<<<<<<<<
 *                 num_consecutive_same_value = 0
 *                 for j in range(s, e):
 * ```

@WillAyd WillAyd mentioned this issue Nov 11, 2023
Merged
@jorisvandenbossche jorisvandenbossche modified the milestones: 2.1.3, 2.1.4 Nov 13, 2023
@lithomas1
Copy link
Member Author

@jorisvandenbossche

Do you want to announce this release and close the issue?
The conda-forge builds should be in now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Release
Projects
None yet
Milestone
2.1.3
Development

No branches or pull requests
3 participants
@WillAyd @jorisvandenbossche @lithomas1