Support JSON mode

[vdjeric]

As the title says, encoder chokes while encoding this file:

{"foo":1}

Original, unreduced input file:

for (;;);{"__ar":1,"payload":null,"bootloadable":{},"ixData":{},"gkxData":{},"lid":"6555274907946641781"}

Encoder stack:

vdjeric-mbp:binjs-ref vdjeric$ ./target/debug/binjs_encode -i ~/breaker2.js -o .
Treating "/Users/vdjeric/breaker2.js" ("")
Output: ./breaker2.binjs
Parsing.
ERROR 2018-05-14T04:01:08Z: binjs::source::shift: Script error { Error: [1:11]: Unexpected token ":"
    at new JsError (/Users/vdjeric/Code/binjs-ref/node_modules/shift-parser/dist/tokenizer.js:165:104)
    at GenericParser.createError (/Users/vdjeric/Code/binjs-ref/node_modules/shift-parser/dist/tokenizer.js:296:14)
    at GenericParser.createUnexpected (/Users/vdjeric/Code/binjs-ref/node_modules/shift-parser/dist/tokenizer.js:275:23)
    at GenericParser.consumeSemicolon (/Users/vdjeric/Code/binjs-ref/node_modules/shift-parser/dist/parser.js:214:20)
    at GenericParser.parseStatementHelper (/Users/vdjeric/Code/binjs-ref/node_modules/shift-parser/dist/parser.js:632:18)
    at GenericParser.isolateCoverGrammar (/Users/vdjeric/Code/binjs-ref/node_modules/shift-parser/dist/parser.js:1084:23)
    at GenericParser.parseStatement (/Users/vdjeric/Code/binjs-ref/node_modules/shift-parser/dist/parser.js:574:23)
    at GenericParser.parseStatementListItem (/Users/vdjeric/Code/binjs-ref/node_modules/shift-parser/dist/parser.js:567:23)
    at GenericParser.parseBlock (/Users/vdjeric/Code/binjs-ref/node_modules/shift-parser/dist/parser.js:1025:24)
    at GenericParser.parseBlockStatement (/Users/vdjeric/Code/binjs-ref/node_modules/shift-parser/dist/parser.js:646:51)
  index: 10,
  line: 1,
  column: 11,
  parseErrorLine: 1,
  parseErrorColumn: 11,
  description: 'Unexpected token ":"' }
[stdin]:15
            /* rethrow */ throw ex;
                          ^

Error: [1:11]: Unexpected token ":"
    at new JsError (/Users/vdjeric/Code/binjs-ref/node_modules/shift-parser/dist/tokenizer.js:165:104)
    at GenericParser.createError (/Users/vdjeric/Code/binjs-ref/node_modules/shift-parser/dist/tokenizer.js:296:14)
    at GenericParser.createUnexpected (/Users/vdjeric/Code/binjs-ref/node_modules/shift-parser/dist/tokenizer.js:275:23)
    at GenericParser.consumeSemicolon (/Users/vdjeric/Code/binjs-ref/node_modules/shift-parser/dist/parser.js:214:20)
    at GenericParser.parseStatementHelper (/Users/vdjeric/Code/binjs-ref/node_modules/shift-parser/dist/parser.js:632:18)
    at GenericParser.isolateCoverGrammar (/Users/vdjeric/Code/binjs-ref/node_modules/shift-parser/dist/parser.js:1084:23)
    at GenericParser.parseStatement (/Users/vdjeric/Code/binjs-ref/node_modules/shift-parser/dist/parser.js:574:23)
    at GenericParser.parseStatementListItem (/Users/vdjeric/Code/binjs-ref/node_modules/shift-parser/dist/parser.js:567:23)
    at GenericParser.parseBlock (/Users/vdjeric/Code/binjs-ref/node_modules/shift-parser/dist/parser.js:1025:24)
    at GenericParser.parseBlockStatement (/Users/vdjeric/Code/binjs-ref/node_modules/shift-parser/dist/parser.js:646:51)

thread 'large stack dedicated thread' panicked at 'Could not parse source: ReturnedError(ExitStatus(ExitStatus(256)))', libcore/result.rs:945:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.
thread 'main' panicked at 'Error in dedicated thread: Any', libcore/result.rs:945:5

[dominiccooney]

Per ExpressionStatement grammar, that is parsed as a block, not an expression statement, hence : is a syntax error.

It looks to me that the original file is actually JSON with an eval guard for(;;);. So the question is more: should binast have a JSON mode? If so, how do you JSON.parse it?

[Yoric]

It looks to me that the original file is actually JSON with an eval guard for(;;);. So the question is more: should binast have a JSON mode? If so, how do you JSON.parse it?

I would actually like it, if compression makes it useful.

[Yoric]

Now that we are using Entropy, we know that we could compress JSON more efficiently than regular JS. See also #296 for the compression side of things.