JSON format issue, especially with larger reports, such as 500MB in size

[pavankumardmg]

👓 What did you see?

When I use JSON format to generate results, I encounter the following error.

error logs:

/app/node_modules/@cucumber/cucumber/src/formatter/json_formatter.ts:185 this.log(JSON.stringify(features, null, 2)) ^ RangeError: Invalid string length at JSON.stringify () at JsonFormatter.onTestRunFinished (/app/node_modules/@cucumber/cucumber/src/formatter/json_formatter.ts:185:19) at EventEmitter. (/app/node_modules/@cucumber/cucumber/src/formatter/json_formatter.ts:92:14) at EventEmitter.emit (node:events:532:35) at EventEmitter.emit (node:domain:488:12) at Coordinator.onWorkerProcessClose (/app/node_modules/@cucumber/cucumber/src/runtime/parallel/coordinator.ts:185:29) at ChildProcess. (/app/node_modules/@cucumber/cucumber/src/runtime/parallel/coordinator.ts:146:12) at ChildProcess.emit (node:events:520:28) at ChildProcess.emit (node:domain:488:12) at maybeClose (node:internal/child_process:1105:16)

✅ What did you expect to see?

Expected to see a JSON report without any formatting issues

📦 Which tool/library version are you using?

node version 18.17.0 or latest cucumber ts 10.6.0

🔬 How could we reproduce it?

Steps to Reproduce:

1. In the runner file, use JSON format and add the JSON path to generate the reports.
2. Include additional screenshots in the test scripts to increase the report size to over 500MB, then observe the json formatter issue after the test is completed.
3. Observe the following error and the same issue has been logged and closed. reference [https://github.com/cucumber/common/issues/1763]

📚 Any additional context?

No response

[mpkorstanje]

The JSON formatter you are using is not able to write streaming data. So total size of the report will always be limited.

We do have a message formatter that does stream the data and writes .ndjson files. Size limitations should be less of a factor here. See https://github.com/cucumber/messages for the format.

Though there are not many tools that can consume this format it yet and we are still migrating our internal tools. This is a slow process and will take time. Currently a few of our internal tools do support it:

• https://github.com/cucumber/html-formatter
• https://github.com/cucumber/junit-xml-formatter
• https://github.com/cucumber/testng-xml-formatter

Though worth noting that the html-formatter does not externalize files. So size issues will come back when viewing the report (https://github.com/cucumber/html-formatter/issues/281).

As Cucumber is an Opensource project I reckon your options are:

• Reduce the size of your reports.
• Requests that your reporting tools support the message format.
• Provide a pull request with that support to your favorite reporting tool.

@davidjgoss is anything people could do to help out with externalizing files in the html formatter?

Note: error message aside, https://github.com/cucumber/common/issues/1763 seems unrelated.

[pavankumardmg]

Hi, Korstanje Thank you for your reply and suggestions. I will definitely check out the options you've suggested.

[davidjgoss]

This has a lot of overlap with the HTML formatter, where also there is a desire to externalise attachments due to their large size bundled into one file.

Per https://github.com/cucumber/html-formatter/issues/281#issuecomment-1899935418:

I think we could also add such an option for the JSON formatter in cucumber-js.

Does that sound reasonable @pavankumardmg?

[pavankumardmg]

Hi @davidjgoss

Thank you for your comments. Yes that sounds good.