Record and replay block traces at runtime

[shuaichang]

Current trace based prefetch happens at build time that requires running a dummy container using the image, adding an accelerated layer and push to a new image. This approach makes using it difficult in the following ways:

1. It changed the semantic of an image to an application. Now instead of many applications using a single image, each application needs to use their own image. The application-level image is quite intrusive to the runtime, e.g.g we need to ask different teams to update their workload to use different images that was actually the same targzip image and it seems unconventional in the container ecosystem.
2. It is difficult to have an accurate trace built during the build time. In large organizations, the runtime environment can be quite complex (many dependencies on database, cloud resources, internal service, different networking, firewalls), it's super hard and costly to have such environment in the build time (which only has Docker etc. but no K8s or any other dependencies).

My thinking of the trace is at application level instead of at image level, it's better to be maintained by the workload/application owners instead of as part of an image. They decide when to record/replay. The interface could be:

• To record:

1. At runtime, the application owner puts a lock file or some other means (e.g. a new binary overlaybd-record {image}) to start recording, the input contains the trace filename
2. Overlaybd starts to record the traces for the image once received such signal
3. Stopping recording the trace could be time or signal based, the output will be an trace file
4. Application owner collects and store the trace file

• To replay

1. Application owner simply put the trace file to a configured location or call a binary overlaybd-replay {image} etc

I'm totally open to suggestion and discussion, I think the trace based prefetch is a super awesome feature and would love to adopt/contribute to make it even better/easier for adoption, thanks!

cc @lihuiba @liulanzheng @BigVan

[liulanzheng]

@beef9999 cc

[yuchen0cc]

#183 We support triggering record and replay process by passing labels through --snapshotter-label when run container https://github.com/containerd/accelerated-container-image/blob/main/docs/trace-prefetch.md#example-usage @shuaichang please have a try.