the new JSON runs out of memory even when streaming

[mcavage]

~~/public/json is just an asset I have of #master . running a simple job in manta I can repro this:

$ mfind -t o /poseidon/stor/logs/muskie/2014/02/04/18 | mjob create -s ~~/public/json -m '/assets/mark.cavage/public/json -gaC "this.audit && this.res.statusCode === 499" -o json-0' -r bunyan -o
mjob create: error: job 3bce4a36-1bac-eb89-ec81-dde8245a128e had 1 error
$ mjob errors 3bce4a36-1bac-eb89-ec81-dde8245a128e | json stderr | xargs mget
FATAL ERROR: JS Allocation failed - process out of memory

[trentm]

FWIW, it is failing on the biggest input:

$ mfind -t o /poseidon/stor/logs/muskie/2014/02/04/18 | xargs mls -l
-rwxr-xr-x 1 poseidon     117296986 Feb 05 20:48 02d02889.log
-rwxr-xr-x 1 poseidon     104721698 Feb 05 20:48 39adec6c.log
-rwxr-xr-x 1 poseidon     129745070 Feb 05 20:48 7854e6a4.log
-rwxr-xr-x 1 poseidon      95889884 Feb 05 20:48 c84b3cab.log
-rwxr-xr-x 1 poseidon      95897406 Feb 05 20:48 60771e58.log
-rwxr-xr-x 1 poseidon      99051533 Feb 05 20:48 1a1ff4e5.log

[trentm]

@mcavage I get the same out of mem error with older jsons:

trent.mick@manta # cat $MANTA_INPUT_FILE | ./json3 -c 'this.audit && this.res.statusCode===409' -o json-0
FATAL ERROR: JS Allocation failed - process out of memory
trent.mick@manta # ./json3 --version
json 3.2.0


trent.mick@manta # cat $MANTA_INPUT_FILE | ./json7 -gac 'this.audit && this.res.statusCode===499' -o json-0 >t
FATAL ERROR: JS Allocation failed - process out of memory
trent.mick@manta # cat $MANTA_INPUT_FILE | ./json6 -gac 'this.audit && this.res.statusCode===499' -o json-0 >t
FATAL ERROR: JS Allocation failed - process out of memory
trent.mick@manta # cat $MANTA_INPUT_FILE | ./json5 -gac 'this.audit && this.res.statusCode===499' -o json-0 >t
FATAL ERROR: JS Allocation failed - process out of memory
trent.mick@manta # cat $MANTA_INPUT_FILE | ./json4 -gac 'this.audit && this.res.statusCode===499' -o json-0 >t
FATAL ERROR: JS Allocation failed - process out of memory

You said in mib@ that "[19:30:29] that's streaming - and the old one does not run out of memory". Can you show an "old one" that does not run out of mem?

[trentm]

Think I found it. json only does streaming mode if all of:

• '-g'
• '-a'
• the input is adjacent objects (adjacent arrays aren't supported for this)
• the output mode is not "json"

Problem is here that you've specified json output... which tells json that it needs to emit actual valid JSON rather that a json stream of separate objects.

So the workaround is:

$ mfind -t o /poseidon/stor/logs/muskie/2014/02/04/18 | mjob create -s /trent.mick/public/bin/json7 -m '/assets/trent.mick/public/bin/json7 -gaC "this.audit && this.res.statusCode === 499" -o jsony-0' -r bunyan -o
added 6 inputs to 6de33b8d-e532-c4b7-9d65-ea9ec9d844cb
[2014-02-04T18:18:03.202Z]  INFO: muskie/HttpServer/38976 on 02d02889-cd80-4ac1-bc0c-4775b86661e4: handled: 499 (audit=true, _audit=true, operation=getpublicstorage, billable_operation=GET
...

specifically that is "jsony-0" output. Yah this is subtle. Not sure the best behaviour. Thoughts:

• docs with streaming examples
• have json support streaming out 'json' output: emit '[', then each object, then ']'. Though that would have broken your case here anyway (bunyan would have balked at the single input array).
• more obvious 'jsonstream' output mode
• 'jsony' is the default mode, but there isn't a way to say '0' indent. perhaps add a '-0' shortcut for 'jsony-0' output mode. So something like json -ga0c COND

[trentm]

Adding -0 option (the last thought) in json v9.