services/horizon: JSON decoder out of sync - data changing underfoot?

[bartekn]

What version are you using?

2.11.0

What did you do?

I've noticed several JSON decoder out of sync - data changing underfoot? errors. After analyzing stack traces I think it happens only in endpoints that implement custom Scanners on scanned DB values (ex. /assets and /liquidity_pools), example:

Stack trace

*errors.errorString JSON decoder out of sync - data changing underfoot?
encoding/json/decode.go:363 (0x64e93e)
encoding/json/decode.go:180 (0x64e21e)
encoding/json/decode.go:107 (0x64dd25)
github.com/stellar/go/services/horizon/internal/db2/history/liquidity_pools.go:48 (0xcc69bd)
database/sql/convert.go:385 (0x4f23e2)
database/sql/sql.go:3246 (0x5028a6)
github.com/jmoiron/[email protected]/sqlx.go:957 (0x69beb2)
github.com/jmoiron/[email protected]/sqlx_context.go:61 (0x69c8df)
github.com/jmoiron/[email protected]/sqlx_context.go:270 (0x69dcfa)
github.com/stellar/go/support/db/session.go:337 (0x90907a)
github.com/stellar/go/support/db/session.go:320 (0x908edf)
github.com/stellar/go/support/db/metrics.go:400 (0x90493a)
github.com/stellar/go/services/horizon/internal/db2/history/liquidity_pools.go:182 (0xcc8337)
github.com/stellar/go/services/horizon/internal/actions/liquidity_pool.go:143 (0xd66898)
github.com/stellar/go/services/horizon/internal/actions/liquidity_pool.go:134 (0xd66754)
github.com/stellar/go/services/horizon/internal/httpx/handler.go:199 (0xdfd893)
github.com/stellar/go/services/horizon/internal/httpx/handler.go:270 (0xdfe431)
net/http/server.go:2046 (0xe0231c)
net/http/server.go:2046 (0xe022fc)
net/http/server.go:2046 (0x89906f)
github.com/go-chi/[email protected]+incompatible/chain.go:31 (0x9d7d0c)
github.com/go-chi/[email protected]+incompatible/mux.go:425 (0x9da7b9)
net/http/server.go:2046 (0x89906f)
github.com/go-chi/[email protected]+incompatible/mux.go:70 (0x9d878e)
github.com/go-chi/[email protected]+incompatible/mux.go:292 (0x9d9fbb)
net/http/server.go:2046 (0x89906f)
github.com/go-chi/[email protected]+incompatible/chain.go:31 (0x9d7d0c)
github.com/go-chi/[email protected]+incompatible/mux.go:425 (0x9da7b9)
net/http/server.go:2046 (0x89906f)
net/http/server.go:2046 (0xe02958)
net/http/server.go:2046 (0xe02950)
net/http/server.go:2046 (0x89906f)
github.com/stellar/[email protected]+incompatible/http.go:73 (0xd7e9e6)
net/http/server.go:2046 (0x89906f)
github.com/rs/[email protected]/cors.go:190 (0xdf91ac)
net/http/server.go:2046 (0x89906f)
github.com/go-chi/[email protected]+incompatible/middleware/compress.go:190 (0xdad88b)
net/http/server.go:2046 (0x89906f)
github.com/stellar/go/services/horizon/internal/httpx/middleware.go:228 (0xe011e2)
net/http/server.go:2046 (0x89906f)
github.com/stellar/go/services/horizon/internal/httpx/middleware.go:111 (0xdfff8f)
net/http/server.go:2046 (0x89906f)
github.com/stellar/go/services/horizon/internal/httpx/middleware.go:84 (0xdffa6e)
net/http/server.go:2046 (0x89906f)
github.com/stellar/go/support/http/xff_middleware.go:51 (0xdfcaaa)
net/http/server.go:2046 (0x89906f)
github.com/stellar/go/services/horizon/internal/httpx/middleware.go:48 (0xdff233)
net/http/server.go:2046 (0x89906f)
github.com/go-chi/[email protected]+incompatible/middleware/request_id.go:76 (0xdb0576)
net/http/server.go:2046 (0x89906f)

I found interesting sentence in Scanner documentation:

	// Reference types such as []byte are only valid until the next call to Scan
	// and should not be retained. Their underlying memory is owned by the driver.
	// If retention is necessary, copy their values before the next call to Scan.

Does it mean only a single Scan can be called by all opened DB sessions?