Big file as request body

[Myriad-Dreamin]

I'm uploading a file as request body with size of 4GB and found that resty copies full file content into memory (to handle http.Request.GetBody callback?). However, it is not necessary because net/http.Client only views the body as an io.ReadCloser.

https://github.com/go-resty/resty/blob/7dda6159eceee10c6cacc6f9d1637042f67423b0/middleware.go#L521-L543

After reading closed issue and full code of resty, no api can help me avoid the http client reading all file content. It is expected that resty can expose a stream-like api or handle a cloneable reader implicitly to get request retrying functionality.

    client.R().SetClonableBody(reqBody.Reader)

or

    sink, err := detectBodyCloner(r) // replace getBodyCopy
    if err != nil && err != ErrBodyIsNotCloneable {
        return err
    }

    var bodyIsNotCloneable = err == ErrBodyIsNotCloneable

    // assign get body func for the underlying raw request instance
    r.RawRequest.GetBody = func() (io.ReadCloser, error) {
        if sink != nil {
            cReader := sink.Clone()
            if reader, ok := cReader.(io.Reader); ok {
                return ioutil.NopCloser(reader), nil
            }
            if reader, ok := cReader.(io.ReadCloser); ok {
                return reader, nil
            }
            return nil, ErrBodyIsNotStream
        }
        if bodyIsNotCloneable {
            return nil, ErrBodyIsNotCloneable
        }
        return nil, nil
    }

The second approach is better in the case of net/http.Client never calling GetBody callback

[kklinan]

I need it, too. I currently do this by storing the file and then streaming the read.

[florentgluck]

I need the ability to upload files of many GB, thus a stream-like implementation is mandatory. @kklinan could you please post your fix here? Thanks!

[gospider007]

this can help you: https://github.com/gospider007/requests/blob/master/test/request/file_test.go

[pcfreak30]

I have just run into this and will have to jump libraries—a bit of a shame. Trying to copy an io.Reader into memory vs streaming is a really bad assumption.

I looked, and I think both https://github.com/monaco-io/request and https://github.com/imroc/req handle this correctly.