pino fails to emit parsable json on large blocks of text

[cdaringe]

Problem

// bug.js
const pino = require("pino");
const logger = pino();
const bigOlString = (c) => Array.from(Array(100)).join(c);

const logme = {
  a: {
    b: {},
    c: Array.from(Array(500)).map((_, i) =>
      bigOlString(String.fromCharCode(128))
    ),
  },
};

logger.info(logme);
//console.log(JSON.stringify(logme));

$ node bug.js | jq . 
# parse error: Unfinished string at EOF at line 1, column 65536

Swap logger.info(logme) with console.log(logme), and parseable content is emitted.

Maybe jq is the problem here

Indeed. Perhaps so. I can't say with certainty. It is technically jq that is being uncool parsing something around the 16 bit max mark. But, the fact that it has no problem with console output suggests that whatever pino serializer is being used may be doing something ...undesirable?

[jsumners]

Did you write it to a file to determine that it is written correctly?

node foo.js | tee output.json

[mcollina]

I tried

node bug.js > out.json
jq . < out.json

and it does not exhibit the problem.

However, it looks like node bug.js | cat - | jq . has the same problem.

It seems jq is not liking how sonic-boom is splitting the data out: https://github.com/mcollina/sonic-boom/blob/1178b1446aef498c76b1cdc4b69d61c794956b71/index.js#L297-L319. I'm inclided to say it's a jq problem because the problem show itself through cat -.

[cdaringe]

What prompted this bug was a real-life usage where at the the 16 bit max mark, two chars (at least as rendered by editor) were dropped. Based on that failure + the above failure, the fact that console is behaving, and that we know sonic boom is doing special processing at a factor of 16, I'd say that sonic-boom is seeming like the most likely candidate. is this a lossy line?

[mcollina]

What do you mean?

It's totally possible that a char is lost somewhere in the process. I find it weird that it works through different destinations.

Would you like to make some test and send a PR?

[Fdawgs]

@cdaringe Did you manage to resolve this?

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.