go
go copied to clipboard
json-iterator
Marshaling slice of structs with one NaN value in the first struct fails slowly vs. encoding/json, also has huge error message
We've discovered an odd performance case in our use of jsoniter. In essence, jsoniter becomes very slow and outputs extremely long error messages when marshaling a slice of structs where the first struct contains a NaN float64. I suspect this generalizes to NaN values "early" in the slice and has something to do with jsoniter's implementation details. Golang's encoding/json fails fast in this situation, as you'll see in the example code.
Is this a bug in jsoniter's implementation? Is it fixable, or a known limitation?
cc @mbolt35 who found this behavior
Test code
package main
import (
"encoding/json"
"fmt"
"math"
"testing"
jsoniter "github.com/json-iterator/go"
)
type Foo struct {
Name string
F float64
}
func makeTestData() []*Foo {
n := 5_000
data := []*Foo{}
for i := 0; i < n; i++ {
f := &Foo{
Name: fmt.Sprintf("foo-%d", n),
F: 43,
}
if i == 0 {
f.F = math.NaN()
}
data = append(data, f)
}
return data
}
func Test_EncodingJSON(t *testing.T) {
data := makeTestData()
bs, err := json.Marshal(data)
if err != nil {
t.Fatal(err)
}
t.Logf("Bs length: %d", len(bs))
}
func Test_Jsoniter(t *testing.T) {
data := makeTestData()
bs, err := jsoniter.ConfigCompatibleWithStandardLibrary.Marshal(data)
if err != nil {
t.Fatal(err)
}
t.Logf("Bs length: %d", len(bs))
}
encoding/json test
→ time go test -v -run 'Test_EncodingJSON'
=== RUN Test_EncodingJSON
nan_test.go:51: json: unsupported value: NaN
--- FAIL: Test_EncodingJSON (0.00s)
FAIL
exit status 1
FAIL github.com/michaelmdresser/play/jsoniternanslow 0.002s
go test -v -run 'Test_EncodingJSON' 0.28s user 0.07s system 220% cpu 0.157 total
go test -v -run 'Test_EncodingJSON' 0.28s user 0.07s system 220% cpu 0.157 total
jsoniter test
The error message is enormous, on the order of the size of the input array for marshaling. I am eliding some of the output with ...etc... so it is readable:
→ time go test -v -run 'Test_Jsoniter'
=== RUN Test_Jsoniter
nan_test.go:60: []*main.Foo: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F:
...etc...
main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: Name: main.Foo.F: unsupported value: NaN
--- FAIL: Test_Jsoniter (0.17s)
FAIL
exit status 1
FAIL github.com/michaelmdresser/play/jsoniternanslow 0.168s
go test -v -run 'Test_Jsoniter' 1.59s user 0.31s system 571% cpu 0.331 total
go test -v -run 'Test_Jsoniter' 1.59s user 0.31s system 571% cpu 0.331 total
The error message gets larger, and the marshaling slower, when the struct has more fields
(Some of the performance hit with more fields may be caused by my terminal emulator rendering more data, but I don't think it accounts for the bulk of the performance hit)
package main
import (
"encoding/json"
"fmt"
"math"
"testing"
jsoniter "github.com/json-iterator/go"
)
type Foo struct {
Name string
Field1 string
Field2 string
Field3 string
F float64
}
func makeTestData() []*Foo {
n := 5_000
data := []*Foo{}
for i := 0; i < n; i++ {
f := &Foo{
Name: fmt.Sprintf("foo-%d", n),
Field1: "test",
Field2: "something goes here",
Field3: "?????",
F: 43,
}
if i == 0 {
f.F = math.NaN()
}
data = append(data, f)
}
return data
}
func Test_EncodingJSON(t *testing.T) {
data := makeTestData()
bs, err := json.Marshal(data)
if err != nil {
t.Fatal(err)
}
t.Logf("Bs length: %d", len(bs))
}
func Test_Jsoniter(t *testing.T) {
data := makeTestData()
bs, err := jsoniter.ConfigCompatibleWithStandardLibrary.Marshal(data)
if err != nil {
t.Fatal(err)
}
t.Logf("Bs length: %d", len(bs))
}
→ time go test -v -run 'Test_EncodingJSON'
=== RUN Test_EncodingJSON
nan_test.go:51: json: unsupported value: NaN
--- FAIL: Test_EncodingJSON (0.00s)
FAIL
exit status 1
FAIL github.com/michaelmdresser/play/jsoniternanslow 0.002s
go test -v -run 'Test_EncodingJSON' 0.28s user 0.04s system 216% cpu 0.146 total
→ time go test -v -run 'Test_Jsoniter'
=== RUN Test_Jsoniter
nan_test.go:60: []*main.Foo: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name:
...etc...
main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: Field3: Field2: Field1: Name: main.Foo.F: unsupported value: NaN
--- FAIL: Test_Jsoniter (0.73s)
FAIL
exit status 1
FAIL github.com/michaelmdresser/play/jsoniternanslow 0.728s
go test -v -run 'Test_Jsoniter' 7.23s user 1.27s system 970% cpu 0.875 total
