mpb icon indicating copy to clipboard operation
mpb copied to clipboard

Add support for ReadSeeker for uploads to s3

Open mskonovalov opened this issue 2 years ago • 5 comments

Unfortunately s3 sdk uses ReadSeeker for uploads. So in this case it is tricky to wrap a file that you trying to upload: curremt API only allows ReadCloser but not ReadSeeker.

mskonovalov avatar Sep 05 '22 08:09 mskonovalov

I never used s3 sdk and probably will not be. IMHO it's horrible approach to use ReadSeeker for uploads. Nevertheless you can wrap ReadCloser returned by (*Bar).ProxyReader and implement whatever you need.

vbauerster avatar Sep 06 '22 07:09 vbauerster

@vbauerster I suspect they use ReadSeeker to be able to re-upload specific parts of the file. That's exactly what I do atm:

ReadSeeker{
	file,
	p.ProxyReader(file),
}

but I suspect this will show wrong results in case you try to re-upload the specific part of the file, it easily can get higher 100%

mskonovalov avatar Sep 07 '22 00:09 mskonovalov

but I suspect this will show wrong results in case you try to re-upload the specific part of the file, it easily can get higher 100%

That's correct. There is SetCurrent which may help to keep bar in correct state after seek has been called. If you come up with working implementation, PR is welcome. I'm not able to test against s3 api.

vbauerster avatar Sep 07 '22 12:09 vbauerster

So I started to test and unfortunately it doesn't work completely: AWS SDK reads body multiple times to calculate different hashes and so on :( Because the reader is seekable they can do it without issues. Don't have ideas yet how to solve it

mskonovalov avatar Sep 14 '22 00:09 mskonovalov

Ok, I was able to come up with some weird naive solution (although it is not working properly in some circumstances):


type ProxyReadSeeker struct {
	F           io.ReadSeekCloser
	ProxyReader io.ReadCloser
	Bar         *mpb.Bar
}

func (rs *ProxyReadSeeker) Read(p []byte) (n int, err error) {
	return rs.ProxyReader.Read(p)
}

func (rs *ProxyReadSeeker) Close() error {
	return rs.F.Close()
}

func (rs *ProxyReadSeeker) Seek(offset int64, whence int) (int64, error) {
	if whence == io.SeekStart {
		rs.Bar.SetCurrent(offset)
	}
	return rs.F.Seek(offset, whence)
}

Not sure if it is useful for anyone else, but it seems to work. Although you cannot rely on automatic completion and have to disable it and complete the bar manually.

mskonovalov avatar Sep 14 '22 02:09 mskonovalov