parse JsonDiff string

[Arnold1]

@valyala hi, How can i parse a json string which looks like that? what is the fastest way to parse it? I data is uint64 id is string debug is uint64 temp is uint64

s := "{\"data\":\"sampleData\", \"id\":\"data_12344\", \"debug\":10000, \"temp\":123213}"

[kevburnsjr]

s := "{\"data\":\"sampleData\", \"id\":\"data_12344\", \"debug\":10000, \"temp\":123213}"

var p fastjson.Parser
v, err := p.Parse(s)
if err != nil {
	log.Fatalf("cannot parse json: %s", err)
}

fmt.Println(string(v.GetStringBytes("id")))
fmt.Println(v.GetInt("debug"))
fmt.Println(v.GetInt("temp"))

// data_12344
// 10000
// 123213

https://play.golang.org/p/VpAhB8arVDk

[Arnold1]

@kevburnsjr @valyala is there a way to make that code better/faster?

I read Data from the json and need to re-write it: https://play.golang.org/p/PJY7bmj6U7Q

I refer to the:

fastjson.Parse(fmt.Sprintf ...
val.Del("Data")
val.Set("Data", dataVal)

the parserPool is the reuse the parser for multiple responses, that why I keep it here...

[kevburnsjr]

This code is a lot more complicated than it needs to be and contains no comments indicating what it is trying to do so let me summarize...

// input
{"Results":[{"ID": 1, "Data": "{\"Calc\":\"0.1\"}}, ...}]}

// output
{"ID": 1, "Data": "{\"Calc\":\"11.2\"}}, {"ID": 2, "Data": "{\"Calc\":\"11.2\"}}, ...

The input contains escaped json.
This program will:

• unmarshal the input
• iterate over results
• unmarshal the escaped json data
• manipulate the Calc value
• marshal the data back to a json string
• update the result
• marshal the result back to json
• write the results to an output buffer

The first step toward optimizing your code is always to write a benchmark test.

Drop this in a file named main_test.go

package main

import (
	"testing"
)

func BenchmarkParsing(b *testing.B) {
	for i := 0; i < b.N; i++ {
		main()
	}
}

Then comment out the fmt.Println in your main function and run :

$ go test --bench=.
BenchmarkParsing-12        84199             14212 ns/op
PASS

I'll be back in a sec with optimizations

[kevburnsjr]

Well, I cleaned it up but it didn't help performance any.
https://play.golang.org/p/Mb78je9twYj

go test -bench=.
BenchmarkBefore-12        157874              7434 ns/op
BenchmarkAfter-12         155823              7651 ns/op
PASS

Every message requires 5 decodes and 4 encodes. There's not really any way to escape that, and 7 µs is probably a lot better than you'll get with any other tool.

I did witness a 2x improvement when the outer parserPool is properly managed. Running this code once you get ~ 14µs while running it many times yields ~7µs.

package main

import (
	"bufio"
	"bytes"
	"testing"
)

func BenchmarkBefore(b *testing.B) {
	e := NewEngineBefore()

	var buf bytes.Buffer
	for i := 0; i < b.N; i++ {
		jsonParser := e.parserPool.Get()
		parsed, _ := e.readResponse(jsonParser, []byte(jsonData))
		e.processResponse(parsed)
		writer := bufio.NewWriter(&buf)
		e.write(parsed, writer)
		writer.Flush()
		e.parserPool.Put(jsonParser)
	}
	_ = buf.Bytes()
}

func BenchmarkAfter(b *testing.B) {
	e := NewEngine()
	var buf bytes.Buffer
	for i := 0; i < b.N; i++ {
		writer := bufio.NewWriter(&buf)
		e.Process([]byte(jsonData), writer)
		writer.Flush()
	}
	_ = buf.Bytes()
}

[Arnold1]

is there a way to optimize setData by optimizing fastjson.Parse(fmt.Sprintf...? oh I see your link above now.

[kevburnsjr]

I replaced that setData Parse with Arena.

var a = &fastjson.Arena{}
val.Set("Data", a.NewStringBytes(data.MarshalTo(nil)))

It didn't really make a difference. fastjson.Parse is very well optimized for use with single values like that.

[Arnold1]

@kevburnsjr

• why do you have 2 parserPool1 *fastjson.ParserPool and how long does fastjson.Arena{} need to be exist?

• lets say I want o set more variables inside "Data". will each variable need its own fastjson.Arena{} or can I do it in the following way?

var a = &fastjson.Arena{}
data.Set("Calc", a.NewNumberFloat64(NewCalc)) // here I update Calc with new float64 value from the calc
data.Set("Meta", a.NewStringBytes(MetaStr)) // here is set MetaStr for DataValue
data.Set("Id2", a.NewStringBytes(Id2Str)) // here is set Id2 for DataValue

val.Set("Data", a.NewStringBytes(data.MarshalTo(nil))) // here I set data value

and will I need to declare all fastjson.Arena before for i, val := range responses ?

• how about using the p2 := e.parserPool2.Get() for fastjson.Parse for parsing all the single values?

a := arenaPool.Get()
defer func() {
    a.Reset()
    arenaPool.Put(a)
}()

data.Set("Calc", a.NewNumberFloat64(NewCalc)) // here I update Calc with new float64 value from the calc
data.Set("Meta", a.NewStringBytes(MetaStr)) // here is set MetaStr for DataValue
data.Set("Id2", a.NewStringBytes(Id2Str)) // here is set Id2 for DataValue

val.Set("Data", a.NewStringBytes(data.MarshalTo(nil))) // here I set data value

• if you use ParserPool or ArenaPool, how often can you call parserPool.Get() or arenaPool.Get() before the for i, val := range responses ? can I share the ParserPool/ArenaPool between different parts of the program and call .Get() + Put()?

[Arnold1]

hi @kevburnsjr did you see my reply?

[kevburnsjr]

parserPool1 is for the entire message, and parserPool2 is for individual data items. I felt that having a separate pool for each message type would probably be more efficient.

As for the other questions, I've devoted about as much time to this thread as I'm willing to.

I've only just begun using this library myself so if you're wondering whether something will work, I'd recommend you do what I do: write some tests and try it out on your own. :wink: