Implement parsing of hexadecimal Unicode characters from JSON #4590
Conversation
antlechner
requested review from
kroening,
peterschrammel and
tautschnig
as code owners
Previously, if a JSON file contained a string in hexadecimal Unicode representation, e.g. "\u0001", the JSON parser would discard the "\u" part and store the string as "0001". This commit fixes this so the resulting string is equal to "\u0001".
antlechner
force-pushed
the
antonia/json-parsing-unicode
branch
from
ff7234d to
06e666b
Compare
| { | ||
| // Character in hexadecimal Unicode representation, in the format | ||
| // \uABCD, i.e. the following four digits are part of this character. | ||
| assert(p + 4 < yyjsontext + len - 1); |
There was a problem hiding this comment.
Suggest p + 5 < yyjsontext + len would be better -- we're checking for 5-byte string uXXXX being in range, after all
There was a problem hiding this comment.
I found the current version to be clearer: when pointing to u, if you advance by four steps (pointing to the last hex character), then you have to still be to the left of the last " of the string. But I'm happy to change it in a follow-up PR if you think your version is more intuitive.
| assert(p + 4 < yyjsontext + len - 1); | ||
| std::string hex(++p, 4); | ||
| result += std::stoi(hex, nullptr, 16); | ||
| p += 3; |
There was a problem hiding this comment.
Comment that you're intentionally leaving p pointing to the last char of the escape
| // \uABCD, i.e. the following four digits are part of this character. | ||
| assert(p + 4 < yyjsontext + len - 1); | ||
| std::string hex(++p, 4); | ||
| result += std::stoi(hex, nullptr, 16); |
There was a problem hiding this comment.
this casts to a char (8-bit uint), so will only work for unicode characters <= 255
There was a problem hiding this comment.
This is addressed in a follow-up PR: #4594
Previously, if a JSON file contained a string in hexadecimal Unicode representation, e.g. "\u0001", the JSON parser would discard the "\u" part and store the string as (the four-character string) "0001". This PR fixes this so the resulting string is equal to (the one-character string) "\u0001".