[Bug-Candidate]: Shrinking extremely slow in some rare situations

[rappie]

Describe the issue:

In some rare situations shrinking can be really slow. I can work around this by setting shrinkLimit really high but it would be nice to have a real fix.

@arcz has access to a private repo where this can be reproduced.

Version:

Many

Relevant log output:

This 1 can shrink to a 0: