colly
colly copied to clipboard
can't get the data of response.Request.Body too
Hello,also I read https://github.com/gocolly/colly/issues/445 and https://github.com/gocolly/colly/issues/438,but i can't read Request.body string.And i try to put it CONTEXT,but it does not work. I look up a lot of information,the 'ioutil.ReadAll' is a probles,it will go to empty the buffer,i also try to 'use ioutil.NopCloser',but it also does not work. There is way to read it,please help me!I'am golang beginner. Looking forward to your reply.
package main
import (
"fmt"
"github.com/gocolly/colly/v2"
"io"
"io/ioutil"
"log"
"net/http"
"strings"
)
func main() {
c := colly.NewCollector()
c.OnResponse(func(r *colly.Response) {
log.Println("resp")
s :=r.Ctx.GetAny("body")
fmt.Println(r.Request.Body)
fmt.Println(s)
reader_bytes,_ :=s.(io.Reader)
body1,_:=ioutil.ReadAll(reader_bytes)
body2,_:=ioutil.ReadAll(r.Request.Body)
fmt.Println("body1:",string(body1))
fmt.Println("body2:",string(body2))
})
c.OnError(func(r *colly.Response, err error) {
log.Println("error:", r.StatusCode, err)
})
c.OnRequest(func(r *colly.Request) {
if r.Body != nil {
r.Ctx.Put("body",r.Body)
}
})
c.Request("POST",
"http://1024game.org/api",
strings.NewReader(`term=4404094910003`),
nil,
http.Header{"Content-Type": []string{"application/x-www-form-urlencoded"}})
}
out:
&{term=4404094910003 18 -1}
&{term=4404094910003 18 -1}
body1:
body2:
After io.Reader
is consumed by the HTTP client, you can't reset it, and will not return any data anymore. io.NopCloser
doesn't help, because it only adds a no-op Close
method. There's no elegant way to work around this, using context is probably the best option. Just store the body string or bytes (not reader!) into context before making request, and retrieve it where you need it.