Optimal Pooling Strategy

[ronag]

What is the optimal strategy when picking the next client to perform a request on in the pool?

I believe the follow sorting criteria would be best:

const next = clients
  .sort((a, b) => {
    // connected
    if (a.connected !== b.connected) return a.connected ? -1 : 1
    // queue size
    if (a.size !== b.size) return a.size - b.size
    // drained timestamp
    if (a.drained !== b.drained) return b.drained - a.drained
    return 0
  })

[ronag]

The more I think about this the more complex of a question this becomes. I'm a little unsure whether we should even have this in this library and if we do where to put the ambition level.

It becomes a trade off between:

• number of connections
• latency
• retries due to closed connections
• timeouts

---

Depth First (current algorithm)

Tries to avoid creating new connections and instead fills existing connections to maximum pipelining factor before creating new ones.

Pros:

• Minimizes number of connections.

Cons:

• Minimal concurrency. Head of line blocking. Small/fast requests get delayed by big/slow requests ahead in the queue.

Breadth first

Pros:

• Minimizes ~~latency~~ head of line blocking.

Cons:

• Maximizes number of connections.

---

Then there are lots of in between that can take different heuristics into account:

• number of existing connections
• request body size
• etc...

One hybrid would be to do breadth first with existing connections (i.e. don't start new ones) and then switch over to depth first.

Other things to keep in mind could be to have different pools for different types of requests:

• Small vs Big body
• idempotent vs nonidempotent
• etc...

Then we also have things such as which connection should we pick from the active pool:

• Most/Least recent activity (lifo, fifo in Node agents)
• etc...

Alas, I'm a little unsure where to put the ambition in terms of this library. Where should we hand over the responsibility to higher level modules/abstractions.

One way could be that we implements depth first and breadth first and then let the user either implement more advanced logic in an abstraction layer above pools or we have some kind of pluggable API.

[ronag]

@szmarczak

[mcollina]

It’s even more complex than this, and I think your description contains something inaccurate.

Depth first minimizes timeouts and maximizes the tcp windows... which in turn minimize latency. However they pay the price of head-of-line blocking in case of a slow request.

Breadth first avoids the problem of head-of-line blocking at the price of higher chance of timeouts and smaller tcp windows.

Unless there is a single request that takes forever, depth first minimize latency..

[ronag]

and I think your description contains something inaccurate.

Updated per your comments.

Depth first minimizes timeouts and maximizes the tcp windows... which in turn minimize latency. However they pay the price of head-of-line blocking in case of a slow request.

Good point.

Breadth first avoids the problem of head-of-line blocking at the price of higher chance of timeouts and smaller tcp windows.

Good point.

Unless there is a single request that takes forever, depth first minimize latency..

Depends on the relative latency of the different requests. If you have a request head of line that takes 10s (for various reasons) in front of a 1s requests... the 1s request will take longer. It's not just about the payload, it also about processing time on the server side, e.g. one request may indirectly call to a database while the other is just in memory on the server side. Plus lots of other factors.

[szmarczak]

Plus lots of other factors.

I agree with @mcollina. I wouldn't worry about those too much. If the following requests are delayed by a huge download (or upload - a user specified timeout), then let the download continue, meanwhile open a new connection for these particular requests and for when the huge download finishes, destroy the connection because there's a new one.