rxvt
Feature Request: Callback Parameters on Download Activities
I would love to have a simple callback parameter which is invoked for every file that is being processed. Something like an on_download optional parameter could be helpful in status tracking or in additional processing.
Use Case 1: Progress Bar Let's say I want to download 1440 CSV files from D3 (one per minute for a day) using a prefix in S3. I'd like to have a callback event for every single file. I might then show a Rich Progress Display in the terminal.
Use Case 2: Pipelining the Download and Compression In the same scenario, I also might add the newly downloaded file name to a memory work queue which is serviced by another thread for adding the file to a compressed tarball. This way, as we are waiting on additional files, we are acting on the files we downloaded right away. Instead of first downloading the files, and then compressing them into an archive serially, we can do this in parallel, saving total work time.
Event Types
The most important kind of event handler in my opinion would be a post-download handler. But there are valid considerations for a pre-download handler too (either passive or governing). Also, we might want to distinguish errors with a failed-download handler.
Event Details Perhaps the following parameters would be helpful:
-
i: index number for this object (ideally in sequential order with a lock counter) -
n: total number of target objects (could be an estimate that refines over time) -
name: s3 object name -
uri: s3 object uri -
status: started, succeeded, failed? -
size: total size in bytes -
timestamp: timestamp for the object last modified -
checksum: s3 checksum -
filename: local filename for the object (if known)
Some of these parameters are meant to be illustrative only. Not all parameters make sense on a pre-download or on a failed-download.
ps. I'm happy to do a quick PR for this feature set -- I'm forking to do this for now, but I'd rather bring back a PR.
Hi Scott, thanks for the FR and PR you raised. The use-case make sense, I'll try to make some time this week to review it and get it merged. Thanks!
