stream-json icon indicating copy to clipboard operation
stream-json copied to clipboard

Add basic simple example of streaming a large JSON file

Open dandv opened this issue 2 years ago • 5 comments

The README for this module is a bit daunting... lots of people are looking simply to parse a large JSON file and get objects one by one:

It would be really helpful to have a simple basic canonical example of how to parse a large JSON file that consists of an arbitrary number of objects in an array. Ideally, that would be a generator pattern, so streaming can be aborted at will, and items iterated through with a for loop like with Python's ijson library:

with open('large_file.json', 'rb') as f:
    objects = ijson.items(f, 'item')
    for o in objects:
        add_object(o)

dandv avatar May 24 '23 22:05 dandv

@uhop: I've added an example at the top of the most upvoted SO answer for the package. What do you think?

dandv avatar May 25 '23 03:05 dandv

It looks good. I would suggest to describe what kind of input is expected. It makes the code more transparent. I plan to put a similar example (or the same one) at the top of README.

PS: make sure that the example actually runs. Non-working examples damage credibility.

uhop avatar May 25 '23 05:05 uhop

It runs fine on an array of objects. I used it in this demo for work.

dandv avatar May 25 '23 06:05 dandv

I tried copying this simple example but it just outputs "undefined" for each JSON element. Why please?

import parser from 'stream-json';
import StreamArray from 'stream-json/streamers/StreamArray.js';
import Chain from 'stream-chain';
import * as fs from "fs";

importUsers().catch(error => console.error(error));

async function importUsers() {
    const users = new Chain([
      fs.createReadStream('users.json'),
      parser(),
      new StreamArray(),
    ]);
    for await (const { user } of users) {
      console.log(user);
    }
}

/*
users.json:

[
  {
    "name": "a"
  },
  {
    "name": "b"
  },
  {
    "name": ""
  }
]

*/

RichardJECooke avatar Jan 19 '24 11:01 RichardJECooke

Never mind, GPT fixed: for await (const { value:user } of users) {

RichardJECooke avatar Jan 19 '24 11:01 RichardJECooke