colly icon indicating copy to clipboard operation
colly copied to clipboard

can't get the data of response.Request.Body too

Open zerokeeper opened this issue 3 years ago • 1 comments

Hello,also I read https://github.com/gocolly/colly/issues/445 and https://github.com/gocolly/colly/issues/438,but i can't read Request.body string.And i try to put it CONTEXT,but it does not work. I look up a lot of information,the 'ioutil.ReadAll' is a probles,it will go to empty the buffer,i also try to 'use ioutil.NopCloser',but it also does not work. There is way to read it,please help me!I'am golang beginner. Looking forward to your reply.

package main

import (
	"fmt"
	"github.com/gocolly/colly/v2"
	"io"
	"io/ioutil"
	"log"
	"net/http"
	"strings"
)


func main() {
	c := colly.NewCollector()

	c.OnResponse(func(r *colly.Response) {
		log.Println("resp")
		s :=r.Ctx.GetAny("body")
		fmt.Println(r.Request.Body)
		fmt.Println(s)
		reader_bytes,_ :=s.(io.Reader)
		body1,_:=ioutil.ReadAll(reader_bytes)
		body2,_:=ioutil.ReadAll(r.Request.Body)
		fmt.Println("body1:",string(body1))
		fmt.Println("body2:",string(body2))

	})

	c.OnError(func(r *colly.Response, err error) {
		log.Println("error:", r.StatusCode, err)
	})

	c.OnRequest(func(r *colly.Request) {
		if r.Body != nil {
			r.Ctx.Put("body",r.Body)

		}
	})
	c.Request("POST",
		"http://1024game.org/api",
		strings.NewReader(`term=4404094910003`),
		nil,
		http.Header{"Content-Type": []string{"application/x-www-form-urlencoded"}})
}

out:

&{term=4404094910003 18 -1}
&{term=4404094910003 18 -1}
body1: 
body2: 

zerokeeper avatar Oct 26 '21 09:10 zerokeeper

After io.Reader is consumed by the HTTP client, you can't reset it, and will not return any data anymore. io.NopCloser doesn't help, because it only adds a no-op Close method. There's no elegant way to work around this, using context is probably the best option. Just store the body string or bytes (not reader!) into context before making request, and retrieve it where you need it.

WGH- avatar Nov 07 '21 17:11 WGH-