boostorg
strings generated by the browser’s JSON.stringify are not parseable by Boost JSON
Some strings generated by the browser’s JSON.stringify are not parseable by Boost JSON in Boost 1.78.0
Browsers can generate JSON strings that look like (CPP string code):
std::string jsonString = "{\"command\":\"[\\uD83D\\uD83D\](file://ud83d/uD83D/)",\"id\":\"id\"}"; // Illegal trailing surrogate
std::string jsonString = "{\"command\":\"[\\uDF3E\\uDEC2\](file://udf3e/uDEC2/)",\"id\":\"id\"}"; // Illegal leading surrogate
std::string jsonString = "{\"command\":\"[\\uD83D\](file://ud83d/)",\"id\":\"id\"}"; // Syntax Error (Half a Surrogate)
The above JSON strings are produced by the following JS when run in a modern browser: JSON.stringify({ command: "\uD83D\uD83D", id: "id" }); JSON.stringify({ command: "\uDF3E\uDEC2", id: "id" }); JSON.stringify({ command: "\uD83D", id: "id" });
This JSON itself is valid UTF-8 strings, but when the strings in the JSON are parsed, and de-escaped the resulting strings are not valid UTF8, and Boost reports parsing errors.
The JSON is produced by browsers as part of the well-formed string specification: https://github.com/tc39/proposal-well-formed-stringify.
The Boost Library provides an option called: boost_parse_options.allow_invalid_utf8 = true; But this has no effect on the output for these strings. Boost still rejects this valid JSON. We also checked with the latest version of Boost and it does not appear this is fixed.
It would be helpful if Boost JSON was able to at least read the other fields, and report an issue for the field(s) that do not parse.
A small program demonstrating the issue: https://godbolt.org/z/eK6xa89x8
So, first of all "\uD83D\uD83D" is not a valid UTF-8 string. It does not represent any sequence of unicode characters. Whatever gives you strings like that does not give you UTF-8 encoded strings. My intuition is that you need to escape the slashes prior to JSON.stringify.
Second, parse_options::allow_invalid_utf8 is intended to disable checking if UTF-8 code units in the text are forming a valid code point. That is, it affects how unescaped UTF-8 is treated.
Now, there's an argument that we need to provide users with a way to allow invalid surrogate sequences in the input. Probably with the same option. The question then becomes, how to deal with such sequences, as they do not represent UTF-8 characters? If we leave them as-is, we ignore the JSON spec, because they are supposedly escaped characters. But we cannot treat them as surrogate pairs, because they do not represent any unicode characters.
If invalid UTF-16 is allowed, unpaired surrogates would probably need to be UTF-8 encoded, and the result will be WTF-8.
Oh, so there is a somewhat popular method of handling it? Would you recommend using a different option for this, or the same allow_invalid_utf8 ?
I think that this deserves a different option, allow_invalid_utf16. The input isn't invalid UTF-8, so it's not the same thing.
Is introducing the new option allow_invalid_utf16 under consideration for upcoming release?
