`json` issues

[arnetheduck]

The json module in the std library has several tricky issues:

• it doesn't actually implement the json standard - instead, it makes up its own notions such as JInt and JFloat neither of which exist in json and neither of which actually cover Number in the standard
• it's incompatible with checked exceptions
• it is buggy in the version we use
• some have pointed out performance issues, thought dealing with these probably is out of scope - it's json after all.

We have the json-serialization library, but it solves a different problem, namely that of mapping Nim types to and from json - it's not so much about implementing json faithfully. It also doesn't implement json numbers.

There's a couple of things we can do:

1. create a new low-level nim-json2 library that implements the standard with "sax" (event based parsing) and "dom" (JsonNode) - maybe based on faststreams, but that increases complexity

• json-serialization would be updated to use this library

2. change the json-serialization library to implement the json standard, then use it in a new nim-json2 library, so as to reuse the low-level parsing
3. copy std/json into stew/shims to get the most critical bugfixes and live with the rest
4. alternatives?

[tersec]

(4) copy std/json into stew/shims to get the most critical bugfixes to unblock 1.2.10, then (1) or (2) or other more long-term alternatives

[arnetheduck]

copy std/json into stew/shims to get the most critical bugfixes

fwiw, I started doing this, but it's good to take a step back and see where it might lead: to a new json library or to keep using upstream json with all its ups and downs - we have a lot of code depending on json, so even just renaming all imports takes a while (I wrote this issue half-way through)

[zah]

The lexer module in nim-json-serialization can already be considered as a small stand-alone SAX-like library for parsing json. nim-json-serialization itself support the parsing of "untyped" json nodes through its support for the JsonString and JsonNode types.

[arnetheduck]

JsonNode suffers the same problems as std/json though, so it's not really a solution (since it is std/json)

[arnetheduck]

https://github.com/status-im/nimbus-eth2/pull/5737 solves this through a rewrite of all relevant code to use nim-json-serialization which has since gained a clean separation between lexer and parser.