Use azure storage V10 to get async storage usage

[toshetah]

This issue is with high importance. We need to upgrade to storage client to V10 in order to get async operations. The current implementation is not async.

In version 10 the Azure Storage Java SDK was separated into 2 sub-modules:

azure-storage-blob
azure-storage-queue

In order to use these modules, they must be added as dependencies to the ingest/pom.xml file, instead of the currently used azure-storage dependency: (Correct to the date of writing this comment)

<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>azure-storage-blob</artifactId>
    <version>10.1.0</version>
</dependency>
<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>azure-storage-queue</artifactId>
    <version>10.0.0-Preview</version>
</dependency>

Note that currently the azure-storage-queue is a preview version

Migrating to version 10 will require a substantial rewrite of any component that interfaces with Azure Storage. Here is a list of the classes in the “ingest” module that are involved with the Azure Storage SDK: (correct to the date of writing this comment)

• AzureStorageClient.java
• IngestClientImpl.java
• IngestionStatus.java
• TableReportIngestionResult.java
• IngestClientImplTest.java
• AzureStorageClientTest.java

[ohadbitt]

Should be followed with performance comparison Note that blob storage v8 does upload to blob blocks multithreaded

[ohadbitt]

#240