erigon icon indicating copy to clipboard operation
erigon copied to clipboard

implement stack based json stream that guarantees valid json in case of errors

Open taratorio opened this issue 1 year ago • 0 comments

Relates to a group of issues related to invalid json responses in our trace_* and debug_* APIs in error cases due to the fact we use json streaming:

  • https://github.com/ledgerwatch/erigon/issues/10376 (more recent)
  • https://github.com/ledgerwatch/erigon/issues/7033 (older, we've had several of these in the past)

Json is not streaming friendly as in case of errors we may end up flushing invalid json with dangling (unclosed) arrays, objects, fields. API implementations need to keep track of these and make sure they correctly close them in case of errors. This is very error prone, difficult and results in messy code.

A simple solution to guarantee that our JSON streams are always valid no matter what is to use a stack based Stream wrapper around jsoniter.Stream:

  • use a Stack to keep track of unclosed "{" and "[" and "unfinished object field writes", "unfinished WriteMore commas", etc.
  • in case of errors at the rpc/handler level call stackStream.ClosePendingObjects which pops the stack and correctly closes nested-ness

Alex mentioned TrueBlocks use a similar approach to handle this problem. We could explore their solution too and potentially re-use if it is easy to.

This way API implementations that use streaming no longer have to worry about closing dangling json in case of errors. They can simply return an error and let the rpc/handler do this.

taratorio avatar May 17 '24 11:05 taratorio