neon icon indicating copy to clipboard operation
neon copied to clipboard

pageserver: separate concurrency limits for big vs. small PUTs

Open jcsp opened this issue 1 year ago • 0 comments

Two reasons:

  • For big puts, there isn't much value in running lots at the same time: assuming a 25gbps link and each flow achieving at least 100mb/s, you don't want more than 25 flows.
  • When the queue for layer uploads is long (it can be tens of seconds long after a big compaction as layers trickle into S3), we should not delay small metadata operations required for timeline creation (uploading index_part.json).

jcsp avatar Dec 19 '23 14:12 jcsp