shrinking memory

[yoshuawuyts]

Currently at no stage do we shrink memory. Ideally we could have a rule running in the background that does something like: "if usage hasn't exceeded 25% in the last 10 seconds, half the amount of items kept in the stack".

We could run this in the background using an async_std::stream::interval inside of a task::spawn kept inside of Pin<Box<JoinHandle>> inside the Pool. We would never await it, and just quietly let it run in the background.

The only downside here is that it ties itself to async-std; but this is probably better than creating a new timer from scratch, spawning a dedicated timer thread, or adding built-in timing logic that's run on each call. I kind of wish this was built into the stdlib so it would be easy to rely on, but alas.

[rylev]

Hmmm, could you explain why you using the timer from async-std is better than a decided timer thread?

I could also imagine a heuristic based on usage instead of time.

[oblique]

Another ideas:

1. We can provide an API for setting a limit for the free queue. If the limit is reached, then we do not push new items.
2. We can use time as @yoshuawuyts proposed, but instead of having a task we can trigger shrinking on Pool::get and ItemGuard::drop.

[yoshuawuyts]

My reasoning for not shrinking based on usage is because it would be hit on every operation. E.g. if we serve a million requests/second, we'd be doing extra (atomic) operations on every one of those. Instead sampling every n millis should provide the same insight at less cost, and checking if we should shrink at most every n seconds is probably the best way to protect against spikes.

But yeah it might very well be that I overestimated these costs, and we notice the overhead of doing this is not big enough that it matters. Probably should test it out (: