easyjson icon indicating copy to clipboard operation
easyjson copied to clipboard

jlexer: Support io.Reader and read in chunks

Open mrVanboy opened this issue 5 years ago • 2 comments

Hi! Thanks for a great library, we are using it a lot in our projects.

What I found today interesting in our profiler, that easyjson.UnmarshalFromReader reads everything from the io.Reader before further processing, which causes huge allocations in case of large data stored in the provided Reader. Yep, it's mentioned in the documentation of the easyjson.UnmarhsalFromReader, but I see here space for improvement.

Profiler screenshot

image

Have you considered reading and parsing chunks of the bytes from io.Reader?

It could help with parsing huge data from net/http.Response.Body as in documentation has mentioned:

The response body is streamed on demand as the Body field is read...

mrVanboy avatar Apr 27 '20 09:04 mrVanboy

Hi! Yes, we did some kind of attempts and found it a little bit difficult to implement immediately :(

shmel1k avatar May 31 '20 18:05 shmel1k

I totally agree, all my allocations are coming from ioutil.ReadAll that I need to call before sending data to easyJson.

I really got happy when I saw UnmarshalFromReader function and but my joy disappeared when I saw its code that uses ioutil.ReadAll too.

mdaliyan avatar Jun 15 '20 09:06 mdaliyan