glaze icon indicating copy to clipboard operation
glaze copied to clipboard

glz::reader/glz::writer for incremental reading/writing

Open stephenberry opened this issue 1 year ago • 7 comments

Glaze doesn't easily support incremental reading and writing. There are times when we want to just parse the first element of an array, perform logic in our program, and then decide how what item to use to parse the rest of the array. This is common for formats that need header information.

Another use case is appending data to a buffer incrementally.

This issue proposes the addition of two stateful classes: gz::reader and glz::writer, which will enable incremental reading/writing.

Initially this will focus on array support, but object support will be considered in the design.

Example syntax:

glz::reader reader{buffer};
reader.read(glz::element<0>(value));
reader.read(glz::rest(value));
glz::writer writer{buffer};
writer.append(value);
write.append(value);

stephenberry avatar May 22 '24 12:05 stephenberry

Addresses: #917

stephenberry avatar May 22 '24 12:05 stephenberry

Note that partial reading, which is supported for JSON, handles this problem in another manner.

stephenberry avatar May 22 '24 20:05 stephenberry

JSON pointer syntax reading also supports this in part. The main difference is that we want a core way to maintain iterators at the parsed location, for the best possible speed. I think this will also take a good step in the direction of handling streams of data.

stephenberry avatar May 22 '24 21:05 stephenberry

The new read_allocated option in #1027 also provides a solution for partial reading.

stephenberry avatar May 23 '24 15:05 stephenberry

read_allocated has been renamed to partial_read in #1028

stephenberry avatar May 23 '24 20:05 stephenberry

The new API design proposal looks nice.

ChristofferGreen avatar Jun 20 '24 07:06 ChristofferGreen

This would be a very welcome addition.

Dooez avatar Oct 09 '24 09:10 Dooez