jsoniter's Number is not strict compliance with JSON RFC 7159 standard

[zhongxinghong]

version

• go version go1.16.7 windows/amd64
• github.com/json-iterator/go v1.1.12
• github.com/stretchr/testify v1.3.0

code to reproduce this issue

package main

import (
	"bytes"
	"encoding/json"
	ejson "encoding/json"
	"testing"

	jsoniter "github.com/json-iterator/go"
	"github.com/stretchr/testify/require"
)

func ijsonDecode(b []byte, pv interface{}) error {
	ijson := jsoniter.ConfigCompatibleWithStandardLibrary
	d := ijson.NewDecoder(bytes.NewReader(b))
	d.UseNumber()
	return d.Decode(pv)
}

func ejsonDecode(b []byte, pv interface{}) error {
	d := ejson.NewDecoder(bytes.NewReader(b))
	d.UseNumber()
	return d.Decode(pv)
}

func TestJSONNumberDiffIE(t *testing.T) {
	var cases = [][]byte{
		[]byte(`{"a": 01}`),
		[]byte(`{"a": 1.}`),
		[]byte(`{"a": 1.e1}`),
		[]byte(`{"a": -.1}`),
	}
	for _, b := range cases {
		t.Logf("case: %v\n", string(b))
		var iv, ev map[string]interface{}
		ierr := ijsonDecode(b, &iv)
		eerr := ejsonDecode(b, &ev)
		t.Logf("ijson value: %v, error: %v\n", iv, ierr)
		t.Logf("ejson value: %v, error: %v\n", ev, eerr)
		require.NoError(t, ierr) // assert ijson accept it
		require.Error(t, eerr)   // assert ejson reject it
		var inum = iv["a"].(json.Number)
		i64, i64err := inum.Int64()
		f64, f64err := inum.Float64()
		t.Logf("ijson Number: i64: %d, i64err: %v", i64, i64err)
		t.Logf("ijson Number: f64: %f, f64err: %v", f64, f64err)
		// assert ijson's Number meets the golang strconv standard
		require.True(t, i64err == nil || f64err == nil)
	}
}

console output

$ go test -v -count=1 json_test.go -run TestJSONNumberDiffIE
=== RUN   TestJSONNumberDiffIE
    json_test.go:34: case: {"a": 01}
    json_test.go:38: ijson value: map[a:01], error: <nil>
    json_test.go:39: ejson value: map[], error: invalid character '1' after object key:value pair
    json_test.go:45: ijson Number: i64: 1, i64err: <nil>
    json_test.go:46: ijson Number: f64: 1.000000, f64err: <nil>
    json_test.go:34: case: {"a": 1.}
    json_test.go:38: ijson value: map[a:1.], error: <nil>
    json_test.go:39: ejson value: map[], error: invalid character '}' after decimal point in numeric literal
    json_test.go:45: ijson Number: i64: 0, i64err: strconv.ParseInt: parsing "1.": invalid syntax
    json_test.go:46: ijson Number: f64: 1.000000, f64err: <nil>
    json_test.go:34: case: {"a": 1.e1}
    json_test.go:38: ijson value: map[a:1.e1], error: <nil>
    json_test.go:39: ejson value: map[], error: invalid character 'e' after decimal point in numeric literal
    json_test.go:45: ijson Number: i64: 0, i64err: strconv.ParseInt: parsing "1.e1": invalid syntax
    json_test.go:46: ijson Number: f64: 10.000000, f64err: <nil>
    json_test.go:34: case: {"a": -.1}
    json_test.go:38: ijson value: map[a:-.1], error: <nil>
    json_test.go:39: ejson value: map[], error: invalid character '.' in numeric literal
    json_test.go:45: ijson Number: i64: 0, i64err: strconv.ParseInt: parsing "-.1": invalid syntax
    json_test.go:46: ijson Number: f64: -0.100000, f64err: <nil>
--- PASS: TestJSONNumberDiffIE (0.00s)
PASS
ok      command-line-arguments  0.071s

discussion

Golang's encoding/json meets the JSON RFC 7159 standard. Its scanner's FSM will check the number literal's grammar first before parsing it by strconv.ParseInt / strconv.ParseFloat, so some number literals that accepted by strconv will be rejected by encoding/json at the first grammar checking pass. But jsoniter does't check it, instead it treats a string that made up of ['+', '-', '.', 'e', 'E', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9'] as a json.Number, so some number literals accepted by golang's strconv package but don't meet the JSON RFC 7159 standard will be wrongly accepted by jsoniter.

Recently I try to replace jsoniter with my custom JSON library that modified from encoding/json to speed up our server, I write many unit-tests to ensure all cases that I could think of will have exactly the same decoding result as the encoding/json. But in our production environment I found some DIFF between jsoniter and my JSON library. The jsoniter's README said that it 100% compatibility with standard lib, but it's actually not. From the above test we can confirm that some strange number literals can be wrongly accepted by jsoniter, which will be rejected by encoding/json. This poses a risk to our JSON library migration because some special cases in our online requests which were fine before will cause errors after migration, and it's hard to prevent it to some extent as we don't know if there are any other DIFF between these two JSON libraries.