AzureSDKForRust
AzureSDKForRust copied to clipboard
Implement streaming operations on blobs
Reading and writing blobs in a single operation is cumbersome and really not efficient. Since we are aync anyway it would make sense to enable data processing as soon as possible without waiting for the complete blob. Also, there might be no need to allocate a big buffer for a file.
The future
crate exposes the Stream
trait so we should implement it.
The read access is completed. It's included in release 0.7.1 merged by #43 .
For example:
let stream = Blob::stream(
&client,
&container_name,
file_name,
None,
&Range::new(0, 4096),
None,
1024,
);
let fut = stream.for_each(move |mut value| {
println!("received {:?} bytes", value.len());
ok(())
});
This will ask the SDK to get the first 4KB of the file in chunks of 1KB. At each chunk you get the chance to do something (power of the future
crate!).
Happy to see this! I will be using it soon. Stoked for the streaming upload as well, as I have already started using the fully buffered upload. Being able to stream the upload will be that much more awesome.
For the streaming upload I need to complete the Put Block REST API first (it was half-baked anyway). This way various chunks of data can be uploaded independently and the finalized with a single Put block list call. Hope to be able to work on this soon!