Performance degradation for streamed content when escaped strings or unicode characters present

[nikolaydimitrov]

We have found that the performance of json.NewDecoder(in).Decode(&out), i.e. when using streamed decoder is very poor. Compared to json.Unmarshal it is about 80-90x times slower.

Profiling shows that memmove is being called too often and is responsible for 96% of the execution. These two places are the culprit:

• https://github.com/goccy/go-json/blob/master/internal/decoder/string.go#L152
• https://github.com/goccy/go-json/blob/master/internal/decoder/string.go#L190

Looking at the code, the problem is that those append (memmove) calls are basically called for every character in the input stream.

[zeroshade]

We ran into the same problem as this in arrow-go, see https://github.com/apache/arrow-go/issues/448