Skip to content

BUG: json.decode fails for nums larger than sys.maxsize #34984

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 19 commits into from
Closed

BUG: json.decode fails for nums larger than sys.maxsize #34984

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
d22da4d
TST: removed xfail from json.decode(long int)
arw2019 Jun 25, 2020
46d05a3
added Object_newBigNum to JSONtoObj.c
arw2019 Jun 25, 2020
cce506c
bug: changed PyLong_FromUnicode to PyLong_FromUnicodeObject
arw2019 Jun 25, 2020
a0a6846
updated JSONObjectDecoder
arw2019 Jun 25, 2020
c60cabc
added case DECODE_BIGNUM to decode_numeric
arw2019 Jun 26, 2020
3c91879
fixed if statement to reset overflowing int
arw2019 Jun 26, 2020
e4eb0af
removed debug statement
arw2019 Jun 26, 2020
f1a3e0a
added cStr variable to PyObjectDecoder
arw2019 Jun 26, 2020
f3894e5
added conversion intValue->string
arw2019 Jun 26, 2020
bfe6595
implemented cStr storage of long int for JSON decoding
arw2019 Jun 27, 2020
c18f685
implemented cStr storage of long int for JSON decoding
arw2019 Jun 27, 2020
4027704
updated ujson header file
arw2019 Jun 27, 2020
e002cc4
updated ujson header file
arw2019 Jun 27, 2020
aa74e5a
styling changes to minimize diff
arw2019 Jun 27, 2020
60cccc8
removed xfail test (fixed in this PR)
arw2019 Jun 27, 2020
6fd68ef
fixed xfail test
arw2019 Jun 27, 2020
74980f4
freed ds->cStr at the end of JSON_DecodeObject
arw2019 Jun 27, 2020
618f97f
TST: updated test
arw2019 Jun 28, 2020
eeea214
fixed Overflow check in ultrajsondec.c
arw2019 Jun 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions pandas/_libs/src/ujson/lib/ultrajson.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -304,6 +304,7 @@ typedef struct __JSONObjectDecoder {
JSOBJ (*newInt)(void *prv, JSINT32 value);
JSOBJ (*newLong)(void *prv, JSINT64 value);
JSOBJ (*newDouble)(void *prv, double value);
JSOBJ (*newBigNum)(void *prv, wchar_t *start, wchar_t *end);
void (*releaseObject)(void *prv, JSOBJ obj, void *decoder);
JSPFN_MALLOC malloc;
JSPFN_FREE free;
Expand Down
14 changes: 9 additions & 5 deletions pandas/_libs/src/ujson/lib/ultrajsondec.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,7 @@ FASTCALL_ATTR JSOBJ FASTCALL_MSVC decode_numeric(struct DecoderState *ds) {
int intNeg = 1;
int mantSize = 0;
JSUINT64 intValue;
JSLONG newDigit;
int chr;
int decimalCount = 0;
double frcValue = 0.0;
Expand Down Expand Up @@ -160,12 +161,15 @@ FASTCALL_ATTR JSOBJ FASTCALL_MSVC decode_numeric(struct DecoderState *ds) {
// FIXME: Check for arithmetic overflow here
// PERF: Don't do 64-bit arithmetic here unless we know we have
// to
intValue = intValue * 10ULL + (JSLONG)(chr - 48);
newDigit = (JSLONG)(chr - 48);

if (intValue > overflowLimit) {
return SetError(ds, -1, overflowLimit == LLONG_MAX
? "Value is too big"
: "Value is too small");
if (intValue*10ULL + newDigit > overflowLimit) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this works generally. If you want a reference for overflow detection try looking at the python implementation of PyLong_AsLongLongAndOverflow

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like this might be ok because overflowLimit is adjusted earlier on depending on whether the number is positive or negative (in line 140 overflowLimit gets changed if intValue came with a minus sign):

pandas/pandas/_libs/src/ujson/lib/ultrajsondec.c

Line 128 in 0159cba

JSUINT64 overflowLimit = LLONG_MAX;

But I'm down to rewrite this part to follow the PyLong_AsLongLongAndOverflow implementation:

https://github.com/python/cpython/blob/04cdeb7a5617c48102f45b965e683b12cdf934f8/Objects/longobject.c#L380

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is if intValue*10ULL + newDigit overflows than the comparison to overflowLimit is a moot point because you've already hit undefined behavior with the former

// TO DO: store current intValue for later processing
// then reset intValue
intValue = (newDigit==0) ? 10 : newDigit;
}
else {
intValue = intValue * 10ULL + newDigit;
}

offset++;
Expand Down
10 changes: 8 additions & 2 deletions pandas/_libs/src/ujson/python/JSONtoObj.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -483,6 +483,12 @@ JSOBJ Object_newDouble(void *prv, double value) {
return PyFloat_FromDouble(value);
}

JSOBJ Object_newBigNum(void* prv, wchar_t *start, wchar_t *end) {
PyObject* obj_as_unicode;
obj_as_unicode = PyUnicode_FromWideChar(start, (end - start));
return PyLong_FromUnicodeObject(obj_as_unicode, 0);
}

static void Object_releaseObject(void *prv, JSOBJ obj, void *_decoder) {
PyObjectDecoder *decoder = (PyObjectDecoder *)_decoder;
if (obj != decoder->npyarr_addr) {
Expand All @@ -509,8 +515,8 @@ PyObject *JSONToObj(PyObject *self, PyObject *args, PyObject *kwargs) {
Object_newPosInf, Object_newNegInf, Object_newObject,
Object_endObject, Object_newArray, Object_endArray,
Object_newInteger, Object_newLong, Object_newDouble,
Object_releaseObject, PyObject_Malloc, PyObject_Free,
PyObject_Realloc};
Object_newBigNum, Object_releaseObject, PyObject_Malloc,
PyObject_Free, PyObject_Realloc};

dec.preciseFloat = 0;
dec.prv = NULL;
Expand Down
3 changes: 1 addition & 2 deletions pandas/tests/io/json/test_ujson.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -568,8 +568,7 @@ def test_dumps_ints_larger_than_maxsize(self, bigNum):
assert str(bigNum) == encoding

# GH20599
with pytest.raises(ValueError):
assert ujson.loads(encoding) == bigNum
assert ujson.decode(encoding) == bigNum

@pytest.mark.parametrize(
"int_exp", ["1337E40", "1.337E40", "1337E+9", "1.337e+40", "1.337E-4"]
Expand Down