[S3 reliability] write accross 2 AZ blobstore implementation

[Arsnael]

The goal would be to offer a feature where we can write blobs across 2 availability zones synchronously.

Configuration changes

In blob.properties

objectstorage.s3.secondary.enabled=true
objectstorage.s3.secondary.endPoint=${env:TMAIL_S3_ENDPOINT}
objectstorage.s3.secondary.region=${env:TMAIL_S3_REGION}
objectstorage.s3.secondary.accessKeyId=${env:TMAIL_S3_ACCESS_KEY}
objectstorage.s3.secondary.secretKey=${env:TMAIL_S3_SECRET_KEY}

Plugged to the Tmail blob module chooser

Code & location

maven module: tmail-baclend/blob/secondary-blob-store

Write a SecondaryBlobStoreDAO class that takes 2 blob store DAO

Nominal case

GIVEN we parallely write to blobStoreA and blobStoreB
WHEN both operation succeeds
THEN we return a storage success

Partial failure

GIVEN we parallely write to blobStoreA and blobStoreB
WHEN write on blobStoreA succeeds and write on blobStoreB fails (or the reverse)
THEN the write fails

The partial error handling modifying this requirement will be handled in an other ticket.

Total failure

GIVEN we parallely write to blobStoreA and blobStoreB
WHEN write on blobStoreA fails and write on blobStoreB fails
THEN the write fails
AND no message is published on RabbitMQ

Read path

Read operation are performed in A, and fallback to B in case of error, or if the object is not found in A.

Plug this into the TMail blob module chooser

Definition of done:

• Unit tests for SecondaryBlobStoreDAO
• Distributed app integration tests: given correctly configured tmail distributed when I receive a mail, it gets stored in A and B