download_object_as_stream for Azure (azure_blobs.py) silently ignores caller's chunk_size without raising an exception.

[AshleyT3]

Summary

The download_object_as_stream for Azure located in azure_blobs.py will silently ignore the caller's specified chunk_size. It should probably raise an exception if the caller's value is not acceptable. Generally, much of chunk-based upload/download seems dependent on knowing all chunks will equal chunk_size until the last chunk. If libcloud silently changes the caller's expectation, it can lead to confusion (see example in details below).

Detailed Information

Below is snippet from azure_blobs.py ...

AZURE_DOWNLOAD_CHUNK_SIZE = (
    int(os.getenv("LIBCLOUD_AZURE_DOWNLOAD_CHUNK_SIZE_MB", "4")) * 1024 * 1024 # 4194304 bytes
)
...
def download_object_as_stream(self, obj, chunk_size=None): 
    """
    @inherits: :class:`StorageDriver.download_object_as_stream`
    """
    obj_path = self._get_object_path(obj.container, obj.name)
    response = self.connection.request(
        obj_path, method="GET", stream=True, raw=True
    )
    iterator = response.iter_content(AZURE_DOWNLOAD_CHUNK_SIZE)
...
drv.download_object_as_stream(storage_object, chunk_size=5*1024*1024) # as one example

In the above example, the caller specified 5MB but immediately receives a smaller chunk given silent chunk_size downgrade by libcloud.