sqewer icon indicating copy to clipboard operation
sqewer copied to clipboard

Introduce SQS connection pooling

Open linkyndy opened this issue 5 years ago • 0 comments

Everytime a new Aws::SQS::Client is instantiated, requests to retrieve SQS credentials are triggered. Since this happens for merely every Sqewer.submit! and derivates, a lot of SQS credential requests are fired, leading to quite a big slowdown.

Here's a small test run with connection pooling:

/app # bundle console
irb(main):001:0> require'./app'
=> true
irb(main):002:0> Net::HTTP.class_eval do alias original_request request; def request(request, body = nil, &block); puts "#{request.method} #{use_ssl? ? "https" : "http"}://#{request["host"] || address}"; original_request(request, body, &block) end end
=> :request
irb(main):004:0> Sqewer.submit!(Job.new(data: 'test'))
GET http://169.254.169.254
GET http://169.254.169.254
POST https://sqs.eu-west-1.amazonaws.com
=> nil
irb(main):005:0> Sqewer.submit!(Job.new(data: 'test'))
POST https://sqs.eu-west-1.amazonaws.com
=> nil
irb(main):006:0> Sqewer.submit!(Job.new(data: 'test'))
POST https://sqs.eu-west-1.amazonaws.com
=> nil
irb(main):007:0> Sqewer.submit!(Job.new(data: 'test'))
POST https://sqs.eu-west-1.amazonaws.com
=> nil

and without connection pooling:

/app # bundle console
irb(main):001:0> require'./app'
=> true
irb(main):002:2" Net::HTTP.class_eval do alias original_request request; def request(request, body = nil, &block); puts "#{request.method} #{use_ssl? ? "https" : "http"}://#{request["host"] || address}"; original_request(request, body, &block) end end
=> :request
irb(main):004:0> Sqewer.submit!(Job.new(data: 'test'))
GET http://169.254.169.254
GET http://169.254.169.254
POST https://sqs.eu-west-1.amazonaws.com
=> nil
irb(main):005:0> Sqewer.submit!(Job.new(data: 'test'))
GET http://169.254.169.254
GET http://169.254.169.254
POST https://sqs.eu-west-1.amazonaws.com
=> nil
irb(main):006:0> Sqewer.submit!(Job.new(data: 'test'))
GET http://169.254.169.254
GET http://169.254.169.254
POST https://sqs.eu-west-1.amazonaws.com
=> nil
irb(main):007:0> Sqewer.submit!(Job.new(data: 'test'))
GET http://169.254.169.254
GET http://169.254.169.254
POST https://sqs.eu-west-1.amazonaws.com
=> nil

We basically start an app console, perform some meta-magic on Net::HTTP to print requests that are being made and then fire a few Sqewer.submit!. You can observe two GET requests to http://169.254.169.254 happening for every submit where connection pooling is not used.

With this PR, there will be a default connection pool available, the same as there was previously a default connection available. The connection pool can also be injected to Sqewer, in a similar way to how a connection would have been previously injected.

linkyndy avatar Jan 24 '20 13:01 linkyndy