tmail-backend icon indicating copy to clipboard operation
tmail-backend copied to clipboard

[PERF] Optimize PublicAsset create by using S3 copy object api

Open vttranlina opened this issue 7 months ago • 3 comments

Why?

ref: https://github.com/linagora/tmail-backend/discussions/1061

Requirement

https://github.com/linagora/james-project/issues/5222

How

Introduce BlobStoreToolBox interface

public interface BlobStoreToolBox {
    Mono<BlobId> copyBlob(BlobId blobId, BucketName sourceBucket, BucketName targetBucket);
}
  • The S3 implement looks like:
public class S3BlobStoreToolBox implements BlobStoreToolBox {
    private final S3AsyncClient s3Client;
    @Override
    public Mono<BlobId> copyBlob(BlobId blobId, BucketName sourceBucket, BucketName targetBucket) {
        return Mono.fromFuture(s3Client.copyObject(CopyObjectRequest.builder()
                .sourceBucket(sourceBucket.asString())
                .sourceKey(blobId.asString())
                .destinationBucket(targetBucket.asString())
                .destinationKey(blobId.asString())
                .build()))
            .publishOn(Schedulers.parallel())
            .thenReturn(blobId);
    }
}
  • For another implement
public class NaiveBlobStoreToolbox implements BlobStoreToolBox {
    private final BlobStore blobStore;
    @Override
    public Mono<BlobId> copyBlob(BlobId blobId, BucketName sourceBucket, BucketName targetBucket) {
        return Mono.from(blobStore.readBytes(sourceBucket, blobId))
            .flatMap(bytes -> Mono.from(blobStore.save(targetBucket, bytes, BlobStore.StoragePolicy.LOW_COST)));
    }
}

Location: tmail-backend/blob/blob-extensions

Adapt PublicAssetSetMethod

Currently, we are using blobResolvers to resolve creationRequest.blobId. I propose that we use UploadRepository directly to retrieve UploadMetaData.

Impact scope:

  • The blobId in the client request must have the prefix uploads-.
  • We can add a new API Publisher<UploadMetaData> listUploads(Username user, UploadId id) to UploadRepository, or use listUploads directly and filter UploadMetaData (in case the upload size for each user is not large).
  • Update the PublicAssetCreationRequest POJO by adding blobId and bucketName field, and removing content field:
case class PublicAssetCreationRequest(size: Size,
                                      contentType: ImageContentType,
                                      identityIds: Seq[IdentityId] = Seq.empty,
                                      blobId: BlobId,
                                      bucketName: BucketName)

Inject BlobStoreToolBox into PublicAssetRepository and use the copyBlob API.

Guice binding

  • using S3BlobStoreToolBox for S3blob store chooser
  • using NaiveBlobStoreToolbox for other remain

vttranlina avatar Jul 16 '24 09:07 vttranlina