got-scraping icon indicating copy to clipboard operation
got-scraping copied to clipboard

Incorrect work on the http2

Open rasimx opened this issue 2 years ago • 3 comments

It seems to me that the library does not work correctly over the http/2 when using a proxy. Each request create a new agent, which, in turn, create of a new connection. I tried to reuse an existing agent in Got, this significantly reduces the time to receive responses from the server.

const http2Agent = new Http2OverHttp(wrapperOptions)

// multiple requests
const response = await got(
    `https://some.site`,
    {
      http2: true,
      resolveBodyOnly: true,
      responseType: 'json',
      agent: {
        http2: http2Agent,
      },
    },
  );

rasimx avatar Oct 20 '23 12:10 rasimx

hi! Taking a look at this issue right now, do you have a reproduction sample we can use to try to debug this? 🙏

Looking at the code, got-scraping only overwrites the agents provided by you when you also set a proxyUrl. Also, what are the options you gave to the Http2OverHttp agent?

vladfrangu avatar Nov 02 '23 15:11 vladfrangu

Apparently, I expressed myself incorrectly. I meant, when using the proxyUrl option, got-scraping creates a new agent for each request. As a result, the new agent creates a new connection for each request. But this does not correspond to the concept of http2, when a new connection is created only at the first request, and all other requests use an existing connection. It should use an existing agent instead of creating a new one for each request. I gave the code above as an example of how I used an existing agent in got

rasimx avatar Nov 03 '23 18:11 rasimx

Please see my comment which explains this issue: https://github.com/apify/got-scraping/issues/112#issuecomment-1969967596

tonybruess avatar Apr 16 '24 20:04 tonybruess