BUG: json.decode fails for nums larger than sys.maxsize

[arw2019]

• closes OverflowError: Python int too large to convert to C long #20599
• tests added
• tests passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

Follow-up to #34473

[arw2019]

@WillAyd I've made some headway on this. I've fixed decode_numeric

pandas/pandas/_libs/src/ujson/lib/ultrajsondec.c

Line 117 in f5b2e5a

FASTCALL_ATTR JSOBJ FASTCALL_MSVC decode_numeric(struct DecoderState *ds) {

so that for intValue, the variable into which we read the incoming int, when it overflows it's reset rather than throwing a ValueError. If I understand this we'd want to store these pre-reset intValues in some way and then combine them into a PyLong object in Object_newBigNum.

Is that right, and if yes what's the best way to go forward? For example, would we want to use a wide_char similarly to what's done for string?

pandas/pandas/_libs/src/ujson/python/JSONtoObj.c

Line 452 in f5b2e5a

JSOBJ Object_newString(void *prv, wchar_t *start, wchar_t *end) {

[WillAyd]

Probably a few approaches here but first that comes to mind is to store the digits in a c string and call PyLong_FromString once all have been read

[WillAyd]

I don't think this works generally. If you want a reference for overflow detection try looking at the python implementation of PyLong_AsLongLongAndOverflow

[arw2019]

I feel like this might be ok because overflowLimit is adjusted earlier on depending on whether the number is positive or negative (in line 140 overflowLimit gets changed if intValue came with a minus sign):

pandas/pandas/_libs/src/ujson/lib/ultrajsondec.c

Line 128 in 0159cba

JSUINT64 overflowLimit = LLONG_MAX;

But I'm down to rewrite this part to follow the PyLong_AsLongLongAndOverflow implementation:

https://github.com/python/cpython/blob/04cdeb7a5617c48102f45b965e683b12cdf934f8/Objects/longobject.c#L380

[WillAyd]

The problem is if intValue*10ULL + newDigit overflows than the comparison to overflowLimit is a moot point because you've already hit undefined behavior with the former

[WillAyd]

No need to change the struct here; just look at the existing implementation of decode_string and mirror that

[arw2019]

@WillAyd Will this work? (intValue is always positive so it's always floor division on the RHS)

[WillAyd]

I don't think so. Assuming a 64 bit platform where the largest integer is 2 ** 63 - 1 this would not branch for 2 ** 63 (which it should)

[arw2019]

Closing for now. I'll open a new request when I've got something ready for review

[kinghuang]

@arw2019 Did you ever create a new PR, by any chance? Thanks to your MR #34473, I'm able to write out large numbers to JSON. But, I can't read them back in.

[arw2019]

@arw2019 Did you ever create a new PR, by any chance? Thanks to your MR #34473, I'm able to write out large numbers to JSON. But, I can't read them back in.

I have an unfinished PR in the works, planning to finish it soon-ish (1.2 is probably a stretch, next release more likely) - unless you'd like to do a PR?