RESS icon indicating copy to clipboard operation
RESS copied to clipboard

Implement TC39 JSON-Superset proposal

Open FreeMasen opened this issue 5 years ago • 1 comments

Allow line separator and paragraph separator in string literals.

proposal

FreeMasen avatar Sep 16 '19 19:09 FreeMasen

The changes to allow for these new characters need to be made in the tokenizer's string method.

Currently there is a check for "is_new_line_not_cr" that will need to be updated to only check for '\n' as the other two characters in that method are now allowed with the JSON superset proposal.

A good test for this would include the following strings:

"\"line\u{2028}seperator\"" "'paragraph\u{2029}seperator'"

The test should correctly evaluate these to the appropriate RawToken and after parsing each the buffer should be empty. A good reference for this is the tokenizer_strings test in the same module.

FreeMasen avatar Nov 06 '19 15:11 FreeMasen