Broken Pipe when sending too many requests

[MagnificentPako]

I'm not sure if reqwest is to be blamed but I'm running into a weird problem where, after sending a bunch of requests, reqwest::get will always return a broken pipe error.

The problem in question happens on these lines: https://github.com/MagnificentPako/rrl-scraper/blob/master/src/main.rs#L73-L76

I put it into a loop because I thought the "broken pipe" error might be a one time thing and it'll get back to it after some retries but it just pauses the scraping process midway.

Any ideas?

[seanmonstar]

I'm not certain what would be causing the broken pipes, but I can suggest one thing: don't use reqwest::get over and over. If you're going to be making multiple requests to the same domain, you'll get better performance creating a Client, since it can pool its connections.

[MagnificentPako]

Thanks for the suggestion! I changed it from using reqwest::get to having a reqwest::Client that's passed around but it still seems to be stuck in the loop after the same amount of scraped chapters.

Here's a backtrace, if that's of any help. https://gist.github.com/MagnificentPako/0c9d4f3a2f198e22579d680bdec9e536

[MagnificentPako]

@seanmonstar bump

Maybe the pooling thing isn't working correctly? Or is it something about my machine not being able to open more sockets, maybe?

[seanmonstar]

It's likely not anything you are doing wrong, and instead a bug in hyper. I have not discovered the root cause yet.

[schell]

@seanmonstar were you ever able to figure this out?

[davemilter]

I use 4 reqwest::Client, each in separate tokio tasks. All take from common query URL and start download via reqwest::Client::get method. All urls for the same sever, I use different clients, to force new http connections, because of there are speed limit for one connection on sever's side.

And time to time I got:

reqwest::Error { kind: Request, url: "...", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(Io, Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" })) }

If I use 1 or 2 Tokio's tasks then in 99.0 percents all works without error.