datamon
datamon copied to clipboard
Integrate progress bar and pipeline and parallelize reads
Reading large files can lead to a poor user experience.
- Integrate with https://github.com/cheggaaa/pb to give better feedback
- look into parallelize CAFS reads and use of io.pipes
Is this for bundle upload and download?
it'd be good to have a proposal for user feedback (that is, to gauge intent of the changes) for large files. cafs parallelization is already done.
https://github.com/oneconcern/datamon/pull/99 and other PRs cover parallelize reads. Without having a clear ask for how to better represent progress, the current CLI works file as the blobs are uploaded and logged in parallel.
there's been an ask from Zhuo for a clearer way to indicate progress on uploads