BUG: Fix buffer overflows in tokenizer.c that caused python to segfault with certain #9360
Conversation
|
whoosh, quite a number of changes for a 'simple' bug..... |
|
there was some pruning of routines that didn't seem to be used anymore and some changes to the way pointers were reallocated and freed while investigating the problem. the heart of the fix is checking lengths against the buffer caps as items are added to the buffers. i thought the extra checking might slow things down slightly, but vbench claims the tokenizer branch is as fast as upstream/master. i'll take a look at the other csv issues you pointed me to earlier |
|
@selasley hahh.. wasn't being critical. I am very glad you refactored. Sometimes just needs to be done! thanks. |
| @@ -210,3 +208,6 @@ Bug Fixes | |||
| - Fixes issue with ``index_col=False`` when ``usecols`` is also specified in ``read_csv``. (:issue:`9082`) | |||
| - Bug where ``wide_to_long`` would modify the input stubnames list (:issue:`9204`) | |||
| - Bug in to_sql not storing float64 values using double precision. (:issue:`9009`) | |||
|
|
|||
|
|
|||
| - Bug in csv. Buffer overflows with certain malformed input files (:issue:`9205`) | |||
|
will have to test this on windows. |
|
:) I was a bit nervous about changing so many lines of other people's code. Thanks for checking on windows. I don't have access to a windows machine at home or work to do any testing. |
|
ok, this looks good on windows. can you do a vbench run on this? just to confirm nothing has changed? ping when done and we'll get this in. |
|
merged via c6c9c0b |
|
thanks! |
closes #9205