jx icon indicating copy to clipboard operation
jx copied to clipboard

ability to stream arbitrary bytes to the underlying writer when in streaming mode

Open elee1766 opened this issue 2 years ago • 1 comments

i have a use case where i would like part of my message to be json streamed encoded from reflection, but i would still like to use jx for the remaining encoding, as most requests are small and do not require any reflection.

currently, when the writer is in streaming mode, all calls to Write will cause an error.

i propose to simply flush the buffer and then pass the call to Write, which would allow now other json-encoders to stream into jx's encoder. i did this in a fork https://github.com/elee1766/jx and it seems to be working?

given that the package does not want to do any reflection based encoding, this seems like a good compromise?

elee1766 avatar Oct 26 '23 10:10 elee1766

example detailing what i cannot figure out how to do

aka, streaming the "result" field of a jsonrpc response.

sometimes these responses can tens of GBS

package main

import (
	"bytes"
	"encoding/json"
	"log"

	"github.com/go-faster/jx"
)

type v struct {
	V string `json:"v"`
}

func main() {

	existingPackage()
	withIoWriter()

}

func existingPackage() {
	val := &v{V: "bar"}
	out := new(bytes.Buffer)
	e := jx.NewStreamingEncoder(out, 4096)

	// CURRENT
	e.Obj(func(e *jx.Encoder) {
		e.Field("jsonrpc", func(e *jx.Encoder) { e.Str("2.0") })
		e.Field("method", func(e *jx.Encoder) { e.Str("method") })
		e.Field("result", func(e *jx.Encoder) {
			// NOTE: in order to encode val, we must first buffer the ENTIRE message
			raw, _ := json.Marshal(val)
			// and pass that in
			e.Raw(raw)
		})
	})
	e.Close()
	ans := out.String()
	log.Println(ans)
	out.Truncate(0)
}
func withIoWriter() {
	val := &v{V: "bar"}
	out := new(bytes.Buffer)
	out.WriteString(`{`)
	out.WriteString(`"jsonrpc":"2.0",`)
	out.WriteString(`"method":"`)
	out.WriteString(`method`)
	out.WriteString(`",`)
	out.WriteString(`"result":`)
	// importantly, this can be streamed directly to network, instead of buffering. also, it adds a newline because it's an encoder, but that is a separate issue :)
	json.NewEncoder(out).Encode(val)
	out.WriteString(`}`)
	ans := out.String()
	log.Println(ans)
	out.Truncate(0)
}

elee1766 avatar Oct 28 '23 21:10 elee1766