Container launch performance test (including registry pull)

[richlander]

Container startup is clearly important. Most of the team is focused on app startup, while registry pull (where size has dominant influence) may be even more important. We should define the contributors to container startup cost and determine which metrics we can and should most affect.

For example, a 1ms app startup win is easy to reason about. It's a 1ms win. How much would we need to reduce container images size by to provide that same 1ms win on container startup? Similarly (but looked at from the opposite angle), how much win do we get from every 1MB of image size reduction?

There are multiple ways to measure this.

• A more basic implementation is with just docker pull on any machine.
• A more sophisticated implementation is with using an actual container service, like AKS or ACI.

Those scenarios both assume no cache hits. It's also reasonable to measure (particularly for a service like AKS) if you have a caching system in place on the host. You might wonder why we'd measure that. What it would show (I'm guessing) is how large an app startup win we need to provide in order for it to be reliably observed. That's super interesting. For example, if we have an 1ms app startup win in ASP.NET Core or CoreCLR, is that within the noise of container startup or reliable observable?

Here's some other things to test:

• Our current offering. That's the baseline.
• Full composite.
• Very minimal R2R (say like 6 assemblies; maybe start with none) and the rest JIT.
• Native AOT + trimming
• Self-contained + trimming

These options establish a nice spectrum of size vs app startup perf options.

We may be able to use our existing TechEmpower related test infra for this, however, it will need to measure a broader timeline of operations than we do today. For some of this (like with AKS), we'll need new infra.

@davidfowl @sebastienros @mangod9 @jkotas