Add basic simple example of streaming a large JSON file
The README for this module is a bit daunting... lots of people are looking simply to parse a large JSON file and get objects one by one:
- SO - Parse large JSON file in Nodejs and handle each object independently (answers recommend this module)
- SO - Parse large JSON file in Nodejs (answers recommend the deprecated JSONstream)
It would be really helpful to have a simple basic canonical example of how to parse a large JSON file that consists of an arbitrary number of objects in an array. Ideally, that would be a generator pattern, so streaming can be aborted at will, and items iterated through with a for loop like with Python's ijson library:
with open('large_file.json', 'rb') as f:
objects = ijson.items(f, 'item')
for o in objects:
add_object(o)
@uhop: I've added an example at the top of the most upvoted SO answer for the package. What do you think?
It looks good. I would suggest to describe what kind of input is expected. It makes the code more transparent. I plan to put a similar example (or the same one) at the top of README.
PS: make sure that the example actually runs. Non-working examples damage credibility.
It runs fine on an array of objects. I used it in this demo for work.
I tried copying this simple example but it just outputs "undefined" for each JSON element. Why please?
import parser from 'stream-json';
import StreamArray from 'stream-json/streamers/StreamArray.js';
import Chain from 'stream-chain';
import * as fs from "fs";
importUsers().catch(error => console.error(error));
async function importUsers() {
const users = new Chain([
fs.createReadStream('users.json'),
parser(),
new StreamArray(),
]);
for await (const { user } of users) {
console.log(user);
}
}
/*
users.json:
[
{
"name": "a"
},
{
"name": "b"
},
{
"name": ""
}
]
*/
Never mind, GPT fixed: for await (const { value:user } of users) {