`.json()` timed out when parsing large response

[Arichy]

I'm fetching a large data in about 36MB:

let response = reqwest::blocking::get(&full_url)?.error_for_status()?;
let body: Value = response.json()?;

And it's easy to get a timed out error

decoding response body: operation timed out

I tried setting the timeout value to 90s, and it spent 87s to deserialize the data to json. It's too slow! As a comparison, I saved the data into a json file and use serde_json to deserialize it directly:

let res: Value = serde_json::from_str(&file)?;

The cost turned out to be about 1s.

[mohe2015]

I think the .json() call includes downloading the response body so maybe your connection is just slow?

[Arichy]

I think the .json() call includes downloading the response body so maybe your connection is just slow?

I'm not sure if the downloading process happens in .get or .json, but my network connection is not so slow as 80s. I used postman and nodejs to send request, and it finished in 5s.

[mohe2015]

Did you try a release build for good measure?

[Arichy]

Sorry, I made a mistake on the request cost. Actually it costed about 20s. I've tested the release build, and the result was between 37 and 56s, still much more than 20s. It's wired because the request costs 20s while the serde_json::from_str(&file) costs 1s, the total cost should be about 21s.

[LeoniePhiline]

Try setting the environment variable RUST_LOG to debug or trace if you are using tracing-subscriber.

This should help you determine where in the code the time is actually spent.