Batch icon indicating copy to clipboard operation
Batch copied to clipboard

Output Files from Batch Task to Azure Blob Storage - be able to set content-MD5 as true, so metadata value is automatically set while uploading blobs

Open kpaliwal0209 opened this issue 1 year ago • 2 comments

Feature Request Description

  • I have Azure Batch Tasks that create many files, and these are uploaded to Azure Blob storage automatically when the task are completed.
  • I'm using the output files to copy these from the node to Azure storage. I was hoping that the [OutputFile](https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.batch.outputfile.-ctor?view=azure-dotnet#microsoft-azure-batch-outputfile-ctor(system-string-microsoft-azure-batch-outputfiledestination-microsoft-azure-batch-outputfileuploadoptions) constructor let you set blob headers like content-md5 as true so the hash of output file is set at runtime and added to blobs metadata. As these are md5 hash value is depended on file content, we can't set to task creation time. I can't see any documentation of this.

Describe Preferred Solution

A clear and concise description of what you want to happen. Make it possible to set content-md5 blob properties as Boolean for output files from an Azure Batch Task as that these values are computed are at runtime and add to the blob metadata.

Describe Alternatives Considered

A clear and concise description of any alternative solutions or features you've considered.

Additional Context

Add any other context or screenshots about the feature request here.

kpaliwal0209 avatar Mar 02 '24 08:03 kpaliwal0209

This is currently possible through the UploadHeaders parameter of OutputFileBlobContainerDestination. Documentation here: https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.batch.outputfileblobcontainerdestination?view=azure-dotnet

staer avatar Mar 02 '24 20:03 staer

Assuming content md5 isn't determinable before hand when you define the output file, this can probably be looked into as a feature request to do it dynamically on upload.

staer avatar Mar 02 '24 20:03 staer