capnproto
Supporting async reads of message segments
While pondering how to handle very large messages, I had the idea that perhaps capnp::message::ReaderSegments::get_segment could return a result that could indicate that the operation is pending, which would allow the implementor of ReaderSegments to asynchronously fetch segments on-demand without blocking on IO. Currently, get_segment only allows for two cases: 1. idx is out of range 2. valid buffer.
If get_segment was to return a 'pending' error, this could potentially be propagated as an error by all reader functions that call it, allowing Reader get() functions to be made into async functions.
Does this seem reasonable, and something you'd like to support eventually?
My initial reaction is that you should try breaking you message into smaller pieces, to avoid the need for this feature.
I'd be curious to hear more about the details of your use case. How many segments do you foresee needing? Where do they need to be fetched from?
Yeah maybe it's a terrible idea. Is it slow to have a lot of segments? I was considering using capnp for game packfiles, so I'd imagine wanting to support >1GB files, probably with more than 100000 segments. You know how with capnp you can usually just mmap a file and then read it? It'd be nice to be able to go from that to some more advanced use-cases.
I'm thinking it could be useful to "hydrate" these files incrementally, downloading segment-by-segment basically. It could also be used to decompress or decrypt segments on-demand.