feat: chunks certification

[3cL1p5e7]

That is a part of proposal about chunks certification (with backward compatibility).

These improvements are a proposal to improve the certification infrastructure around IC and might be considered as a recommendation for dfinity-team.

Goal

Make it possible to certify asset chunks. Validate chunk certificates on the service-worker and icx-proxy.

Why

At the moment, the service-worker and icx-proxy does not support the certification of chunkified files. Moreover, right now it is not possible to correctly stream chunkified and large audio and video files to the front-end. This problems could be solved independently if it would be possible to install an additional service-worker in the certified zone of the domain ic0.app (for 206 partial http-request handling). But is is impossible because there is unable to place custom worker on ic0.app domain. Making your own custom player for audio and video is extremely difficult due to the large number of formats and non-native implementation.

Details

To make this possible, support for HTTP-range requests for http_request query method has been added. This is done to support native html audio/video element (which uses 206 partial http-request) and to determine the index of the chunk throught 206 partial http-request. Using 206 partial http-requests allows you to focus only on certification in the worker and icx-proxy.

Steps

1. It all starts with PR for certified-assets-canister in cdk-rs
2. Did file was updated in PR for certified-assets-canister
3. Service-worker started supporting chunk_tree certificate verification in PR for ic
4. (Here) icx-proxy started supporting chunk_tree certificate verification here
5. Added support for new certified-assets-canister did in PR for agent-rs

[3cL1p5e7]

Tests are not passing due to PR to agent-rs

[Daniel-Bloom-dfinity]

I really think we should separate the unzip change from this. Ideally also the refactors like decode_hash_tree. If you could move those changes into their own PRs separate from this (1 for each would be best), they would be much easier to approve. The chunks change is a protocol change and thus has many more moving parts (like agent-rs changes) and requires more thorough consideration and review.

Really appreciate all this work though!

[3cL1p5e7]

@Daniel-Bloom-dfinity body decoding is separated to another PR

[paulyoung]

Are there still plans to add this feature?