s3fetch
s3fetch copied to clipboard
Added an optional download activity callback for tracking and parallel work processing
I added an optional download activity callback for the run() method. If the on_download in run() is not None, then we do a protected callback to the handler with some context properties (a dictionary with key, s3_uri, size, timestamp, status, and local filename). This enables clients to do helpful things, such as:
- Update a progress bar like Rich
- Start working on each file as it is downloaded
- Manage Ops Reporting
In my case, I want to make a compressed tar archive, and I don't want to wait for it all to be downloaded first.