aioaws icon indicating copy to clipboard operation
aioaws copied to clipboard

MultiPartUpload support

Open MrPigss opened this issue 3 years ago • 2 comments

While this PR should be working but is not yet ready to be merged. No tests have been written. The functionality has only been tested using a Minio s3 bucket.

Any remarks / tips are welcome. Help with mocking the request for testing would be appreciated.

This should implement following api's:

Todo:

  • Fix AWSv4Auth compatibility -> only public buckets work for now
  • write mocks
  • write tests

Example

import asyncio
from aioaws.s3 import S3Client, S3Config
from httpx import AsyncClient


async def s3_demo(client: AsyncClient):
    s3 = S3Client(client, S3Config('<access key>', '<secret key>', '<region>', 'my_bucket_name.com'))

    await s3.delete('path/testt.txt')

    async with s3.createMultipartUpload('path/testt.txt') as mpu:
       
        # These can also be uploaded in parallel using tasks
        await mpu.uploadPart(1, 'abcde' * 1_000_000)
        await mpu.uploadPart(2, 'abcde' * 1_000_000)
        await mpu.uploadPart(2, 'end')  # last part doesn't have to be 5mb

        x = await mpu.listParts(1000)
        print(x)  # should print 3 parts

    async for x in s3.list():
        print(x)  # should contain testt.txt


async def main():
    async with AsyncClient(timeout=30) as client:
        await s3_demo(client)


asyncio.run(main())

MrPigss avatar Feb 16 '23 15:02 MrPigss

Codecov Report

Merging #41 (316909d) into main (332855f) will decrease coverage by 7.25%. The diff coverage is 24.19%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #41      +/-   ##
==========================================
- Coverage   96.64%   89.39%   -7.25%     
==========================================
  Files          10       10              
  Lines         626      679      +53     
  Branches      102      114      +12     
==========================================
+ Hits          605      607       +2     
- Misses         15       64      +49     
- Partials        6        8       +2     
Impacted Files Coverage Δ
aioaws/ses.py 95.73% <ø> (ø)
aioaws/s3.py 72.37% <21.66%> (-26.83%) :arrow_down:
aioaws/core.py 98.34% <100.00%> (-1.66%) :arrow_down:
aioaws/testing.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 332855f...316909d. Read the comment docs.

codecov[bot] avatar Feb 16 '23 15:02 codecov[bot]