tiled icon indicating copy to clipboard operation
tiled copied to clipboard

Support access to raw encoded chunks

Open danielballan opened this issue 3 years ago • 2 comments

I think we need to start thinking about how to let the client elide

read -> decompress -> deserialize -> serialize -> compress -> transport -> decompress -> deserialize

into

read -> transport -> decompress -> deserialize

if the client happens to understand the formats and codec(s) that the data happens to be stored in.

This involves:

  • Accessing a description of the raw encoding, perhaps via a new raw_encoding property on Adapters. We may take inspiration from Zarr regarding how to spell this.
  • Comparing a requests's Accept and Accept-Encoding headers to the raw encoding, and short-circuiting the serialization and (sometimes) the compression process if the client can accept the raw encoding.

danielballan avatar Aug 05 '22 12:08 danielballan

I think there is a clear path to doing this in a backward-compatible way, so I have removed it from the v0.1.0 milestone.

danielballan avatar Oct 14 '22 12:10 danielballan

This could also work for nodes. If I ask for a node that happens to be an HDF5 file as HDF5, we can just spool the HDF5 file.

It might make sense to give the client a way to know this by sticking some information in meta.

danielballan avatar Nov 04 '22 18:11 danielballan