How can I stream JSON and not have elements truncated?

[dandv]

Escalating from discussion in https://github.com/visgl/loaders.gl/discussions/2267

Escalating as an issue since the behavior might be a bug, or expectations could be clarified in the JSON documentation.

The issue is that while streaming an array of key-value objects, of which one value is another 2048-element array of numbers, that array is most often truncated. My expectation was that each array element would be intact.

The array elements are objects with this structure:

{
    "text": "...",
    "values": [
        0,
        1,
        ...
        2047
    ]
},

values is an array of 2048 floats, and text is at most ~500 characters.

The problem is that the values array is most of the time returned incompletely, e.g. with only 1488 elements, even though batch.batchType was data. My expectation was that each root JSON array element would be left intact, i.e. the values array should always contain all 2048 values from the file.

The code I'm using is that from the JSON docs, and detailed in the discussion.

[alexstoken]

Hi @dandv, I think I'm having the same issue. Did you ever find a solution? Thanks!