AzureSDKForRust icon indicating copy to clipboard operation
AzureSDKForRust copied to clipboard

Implement streaming operations on blobs

Open MindFlavor opened this issue 6 years ago • 3 comments

Reading and writing blobs in a single operation is cumbersome and really not efficient. Since we are aync anyway it would make sense to enable data processing as soon as possible without waiting for the complete blob. Also, there might be no need to allocate a big buffer for a file.

The future crate exposes the Stream trait so we should implement it.

MindFlavor avatar May 30 '18 14:05 MindFlavor

The read access is completed. It's included in release 0.7.1 merged by #43 .

For example:

let stream = Blob::stream(
        &client,
        &container_name,
        file_name,
        None,
        &Range::new(0, 4096),
        None,
        1024,
);

let fut = stream.for_each(move |mut value| {
        println!("received {:?} bytes", value.len());
        ok(())
});

This will ask the SDK to get the first 4KB of the file in chunks of 1KB. At each chunk you get the chance to do something (power of the future crate!).

MindFlavor avatar May 30 '18 16:05 MindFlavor

Happy to see this! I will be using it soon. Stoked for the streaming upload as well, as I have already started using the fully buffered upload. Being able to stream the upload will be that much more awesome.

thedodd avatar Jun 04 '18 15:06 thedodd

For the streaming upload I need to complete the Put Block REST API first (it was half-baked anyway). This way various chunks of data can be uploaded independently and the finalized with a single Put block list call. Hope to be able to work on this soon!

MindFlavor avatar Jun 04 '18 20:06 MindFlavor