resty icon indicating copy to clipboard operation
resty copied to clipboard

Limit the maximum number of requests

Open liyuan1125 opened this issue 3 years ago • 5 comments

For example, how to limit the maximum number of requests during concurrent requests?

liyuan1125 avatar Oct 01 '21 03:10 liyuan1125

@leewithyuan I'm sorry, I didn't get it. Do you mean throttling?

jeevatkm avatar Oct 24 '21 23:10 jeevatkm

request pool?

zhaokai5520 avatar Oct 29 '21 07:10 zhaokai5520

Is this a duplicate of https://github.com/go-resty/resty/issues/256?

lavoiesl avatar Oct 29 '21 18:10 lavoiesl

@leewithyuan I'm sorry, I didn't get it. Do you mean throttling?

Sorry, I have solved it, the maximum number of requests is limited to 100; But it would be better if there is a more convenient api;

      var reqChan = make(chan time.Time, 100)

	cli := resty.New()

	cli.OnBeforeRequest(func(client *resty.Client, request *resty.Request) error {
		reqChan <- request.Time
		return nil
	})

	cli.OnAfterResponse(func(client *resty.Client, response *resty.Response) error {
		<-reqChan
		return nil
	})

      for i := 0; i <= 1000; i++ {
		go cli.R().Get("http://xxxxxxx")
	}

liyuan1125 avatar Oct 30 '21 15:10 liyuan1125

I believe the correct way to do this is construct a new http.Transport and give it a MaxConnsPerHost.

Something like this:

dialer := &net.Dialer{
	Timeout:   30 * time.Second,
	KeepAlive: 30 * time.Second,
}
restyClient.SetTransport(&http.Transport{
	Proxy:                 http.ProxyFromEnvironment,
	DialContext:           dialer.DialContext,
	ForceAttemptHTTP2:     true,
	MaxIdleConns:          100,
	IdleConnTimeout:       90 * time.Second,
	TLSHandshakeTimeout:   10 * time.Second,
	ExpectContinueTimeout: 1 * time.Second,
	MaxIdleConnsPerHost:   runtime.GOMAXPROCS(0) + 1,
	MaxConnsPerHost: 100,
})

tylerstillwater avatar Jan 30 '22 01:01 tylerstillwater