mpb
mpb copied to clipboard
Add support for ReadSeeker for uploads to s3
Unfortunately s3 sdk uses ReadSeeker for uploads. So in this case it is tricky to wrap a file that you trying to upload: curremt API only allows ReadCloser but not ReadSeeker.
I never used s3 sdk and probably will not be. IMHO it's horrible approach to use ReadSeeker for uploads. Nevertheless you can wrap ReadCloser returned by (*Bar).ProxyReader and implement whatever you need.
@vbauerster I suspect they use ReadSeeker to be able to re-upload specific parts of the file. That's exactly what I do atm:
ReadSeeker{
file,
p.ProxyReader(file),
}
but I suspect this will show wrong results in case you try to re-upload the specific part of the file, it easily can get higher 100%
but I suspect this will show wrong results in case you try to re-upload the specific part of the file, it easily can get higher 100%
That's correct. There is SetCurrent which may help to keep bar in correct state after seek has been called. If you come up with working implementation, PR is welcome. I'm not able to test against s3 api.
So I started to test and unfortunately it doesn't work completely: AWS SDK reads body multiple times to calculate different hashes and so on :( Because the reader is seekable they can do it without issues. Don't have ideas yet how to solve it
Ok, I was able to come up with some weird naive solution (although it is not working properly in some circumstances):
type ProxyReadSeeker struct {
F io.ReadSeekCloser
ProxyReader io.ReadCloser
Bar *mpb.Bar
}
func (rs *ProxyReadSeeker) Read(p []byte) (n int, err error) {
return rs.ProxyReader.Read(p)
}
func (rs *ProxyReadSeeker) Close() error {
return rs.F.Close()
}
func (rs *ProxyReadSeeker) Seek(offset int64, whence int) (int64, error) {
if whence == io.SeekStart {
rs.Bar.SetCurrent(offset)
}
return rs.F.Seek(offset, whence)
}
Not sure if it is useful for anyone else, but it seems to work. Although you cannot rely on automatic completion and have to disable it and complete the bar manually.