easyjson
easyjson copied to clipboard
jlexer: Support io.Reader and read in chunks
Hi! Thanks for a great library, we are using it a lot in our projects.
What I found today interesting in our profiler, that easyjson.UnmarshalFromReader reads everything from the io.Reader before further processing, which causes huge allocations in case of large data stored in the provided Reader. Yep, it's mentioned in the documentation of the easyjson.UnmarhsalFromReader, but I see here space for improvement.
Profiler screenshot

Have you considered reading and parsing chunks of the bytes from io.Reader?
It could help with parsing huge data from net/http.Response.Body as in documentation has mentioned:
The response body is streamed on demand as the Body field is read...
Hi! Yes, we did some kind of attempts and found it a little bit difficult to implement immediately :(
I totally agree, all my allocations are coming from ioutil.ReadAll that I need to call before sending data to easyJson.
I really got happy when I saw UnmarshalFromReader function and but my joy disappeared when I saw its code that uses ioutil.ReadAll too.