nativelink icon indicating copy to clipboard operation
nativelink copied to clipboard

Add support for compressed blob uploads

Open chrisstaite-menlo opened this issue 2 years ago • 6 comments

Currently we only support compression on the transport, add support for compressed uploads:

main/uploads/94448848-5c26-4e27-9448-50f9fea1947e/compressed-blobs/zstd/33cc97ba1cbeb6c3c542d594f3d96cf0a80f562a287c50f61783763be42586bd/79

chrisstaite-menlo avatar Sep 09 '23 08:09 chrisstaite-menlo

First update remote_execution.proto to v2.3 in #268 Then we can add support for the different compression types

chrisstaite-menlo avatar Sep 11 '23 14:09 chrisstaite-menlo

#281 adds support for reading the compressed blob resource names. Now we just need to handle the compressed blob.

chrisstaite-menlo avatar Sep 14 '23 12:09 chrisstaite-menlo

Might be nice if these were passed through GrpcStore and then if the CompressionStore were in use it could simply used the pre-compressed content?

chrisstaite-menlo avatar Sep 15 '23 16:09 chrisstaite-menlo

Might be nice if these were passed through GrpcStore and then if the CompressionStore were in use it could simply used the pre-compressed content?

We can read the capabilities API in the GrpcStore, and if it supports compressed uploads/downloads use it?

allada avatar Sep 16 '23 06:09 allada

I'd like to work on this issue. cc: @allada

blizzardc0der avatar Apr 04 '24 17:04 blizzardc0der