[Bug]: During epoch, white_whale's contract panic'd

[joeabbey]

What happened?

A recovered panic occurred:

Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: 5:27PM ERR recovered (error) panic: Error parsing into type white_whale_std::pool_network::pair::Config: unknown field `cosmwasm_pool_interface`, expected one of `owner`, `fee_collector_addr`, `pool_fees`, `feature_toggle`: query wasm contract failed module=server
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: 5:27PM ERR stack trace: goroutine 105 [running]:
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: runtime/debug.Stack()
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         runtime/debug/stack.go:24 +0x5e
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/osmosis-labs/osmosis/osmoutils.PrintPanicRecoveryError({{0x3d4aec8, 0x591cf00}, {0x3d66800, 0xc08317fe00}, {{0xb, 0x0}, {0xc07b333402, 0x9}, 0xd7d30e, {0x37df29d0, ...}, ...}, ...}, ...)
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/osmosis-labs/osmosis/[email protected]/cache_ctx.go:61 +0x4a
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/osmosis-labs/osmosis/osmoutils.ApplyFuncIfNoError.func1()
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/osmosis-labs/osmosis/[email protected]/cache_ctx.go:28 +0xb8
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: panic({0x2bb7840?, 0xc0bd49f4a0?})
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         runtime/panic.go:914 +0x21f
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/osmosis-labs/osmosis/osmoutils/cosmwasm.MustQuery[...]({{0x3d4aec8, 0x591cf00}, {0x3d66800, 0xc08426eb00}, {{0xb, 0x0}, {0xc07b333402, 0x9}, 0xd7d30e, {0x37df29d0, ...}, ...}, ...}, ...)
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/osmosis-labs/osmosis/[email protected]/cosmwasm/helpers.go:92 +0x85
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/osmosis-labs/osmosis/v23/x/cosmwasmpool/model.Pool.GetSpreadFactor({{{_, _}, _, _, {_, _, _}}, {_, _}}, {{0x3d4aec8, ...}, ...})
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/osmosis-labs/osmosis/v23/x/cosmwasmpool/model/pool.go:77 +0xd0
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/osmosis-labs/osmosis/v23/x/poolmanager.Keeper.SwapExactAmountInNoTakerFee({{0x3d21660, 0xc00161c560}, {0x3d63418, 0xc001834000}, {0x3d63c20, 0xc0019d47e0}, {0x3d63c90, 0xc001530c30}, {0x3d15fe0, 0xc000178e00}, ...}, ...)
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/osmosis-labs/osmosis/v23/x/poolmanager/router.go:218 +0x1bd
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/osmosis-labs/osmosis/v23/x/txfees/keeper.Keeper.swapNonNativeFeeToDenom.func1({{0x3d4aec8, 0x591cf00}, {0x3d66800, 0xc08426eb00}, {{0xb, 0x0}, {0xc07b333402, 0x9}, 0xd7d30e, {0x37df29d0, ...}, ...}, ...})
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/osmosis-labs/osmosis/v23/x/txfees/keeper/hooks.go:164 +0x13e
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/osmosis-labs/osmosis/osmoutils.ApplyFuncIfNoError({{0x3d4aec8, 0x591cf00}, {0x3d66800, 0xc08317fe00}, {{0xb, 0x0}, {0xc07b333402, 0x9}, 0xd7d30e, {0x37df29d0, ...}, ...}, ...}, ...)
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/osmosis-labs/osmosis/[email protected]/cache_ctx.go:35 +0x13e
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/osmosis-labs/osmosis/v23/x/txfees/keeper.Keeper.swapNonNativeFeeToDenom({{0x3d21660, 0xc00161c580}, {0x3d22ad0, 0xc001680140}, {0x3d55740, 0xc0017a8000}, {0x3d50600, 0xc0019d4900}, {0x3d15ea0, 0xc000178a80}, ...}, ...)
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/osmosis-labs/osmosis/v23/x/txfees/keeper/hooks.go:155 +0x4a6
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/osmosis-labs/osmosis/v23/x/txfees/keeper.Keeper.AfterEpochEnd({{0x3d21660, 0xc00161c580}, {0x3d22ad0, 0xc001680140}, {0x3d55740, 0xc0017a8000}, {0x3d50600, 0xc0019d4900}, {0x3d15ea0, 0xc000178a80}, ...}, ...)
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/osmosis-labs/osmosis/v23/x/txfees/keeper/hooks.go:35 +0x115
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/osmosis-labs/osmosis/v23/x/txfees/keeper.Hooks.AfterEpochEnd(...)
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/osmosis-labs/osmosis/v23/x/txfees/keeper/hooks.go:108
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/osmosis-labs/osmosis/x/epochs/types.panicCatchingEpochHook.func1({{0x3d4aec8, 0x591cf00}, {0x3d66800, 0xc08317fe00}, {{0xb, 0x0}, {0xc07b333402, 0x9}, 0xd7d30e, {0x37df29d0, ...}, ...}, ...})
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/osmosis-labs/osmosis/x/[email protected]/types/hooks.go:50 +0x59
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/osmosis-labs/osmosis/osmoutils.ApplyFuncIfNoError({{0x3d4aec8, 0x591cf00}, {0x3d66800, 0xc08317f8c0}, {{0xb, 0x0}, {0xc07b333402, 0x9}, 0xd7d30e, {0x37df29d0, ...}, ...}, ...}, ...)
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/osmosis-labs/osmosis/[email protected]/cache_ctx.go:35 +0x13e
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/osmosis-labs/osmosis/x/epochs/types.panicCatchingEpochHook({{0x3d4aec8, 0x591cf00}, {0x3d66800, 0xc08317f8c0}, {{0xb, 0x0}, {0xc07b333402, 0x9}, 0xd7d30e, {0x37df29d0, ...}, ...}, ...}, ...)
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/osmosis-labs/osmosis/x/[email protected]/types/hooks.go:53 +0xa5
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/osmosis-labs/osmosis/x/epochs/types.MultiEpochHooks.AfterEpochEnd(...)
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/osmosis-labs/osmosis/x/[email protected]/types/hooks.go:30
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/osmosis-labs/osmosis/x/epochs/keeper.Keeper.AfterEpochEnd(...)
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/osmosis-labs/osmosis/x/[email protected]/keeper/hooks.go:10
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/osmosis-labs/osmosis/x/epochs/keeper.Keeper.BeginBlocker.func1(0xc0b6b18b60?, {{0xc114c00368, 0x3}, {0x0, 0xed85ec810, 0x0}, 0x4e94914f0000, 0x3de, {0x3588d964, 0xedd77f8d9, ...}, ...})
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/osmosis-labs/osmosis/x/[email protected]/keeper/abci.go:46 +0xc63
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/osmosis-labs/osmosis/x/epochs/keeper.Keeper.IterateEpochInfo({{_, _}, {_, _}}, {{0x3d4aec8, 0x591cf00}, {0x3d66800, 0xc08317f8c0}, {{0xb, 0x0}, ...}, ...}, ...)
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/osmosis-labs/osmosis/x/[email protected]/keeper/epoch.go:82 +0x1be
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/osmosis-labs/osmosis/x/epochs/keeper.Keeper.BeginBlocker({{_, _}, {_, _}}, {{0x3d4aec8, 0x591cf00}, {0x3d66800, 0xc08317f8c0}, {{0xb, 0x0}, ...}, ...})
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/osmosis-labs/osmosis/x/[email protected]/keeper/abci.go:16 +0x165
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/osmosis-labs/osmosis/x/epochs.AppModule.BeginBlock(...)
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/osmosis-labs/osmosis/x/[email protected]/module.go:146
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x3d4aec8, 0x591cf00}, {0x3d66800, 0xc08317f8c0}, {{0xb, 0x0}, {0xc07b333402, 0x9}, 0xd7d30e, ...}, ...}, ...)
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/cosmos/[email protected]/types/module/module.go:562 +0x17b
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/osmosis-labs/osmosis/v23/app.(*OsmosisApp).BeginBlocker(_, {{0x3d4aec8, 0x591cf00}, {0x3d66800, 0xc08317f8c0}, {{0xb, 0x0}, {0xc07b333402, 0x9}, 0xd7d30e, ...}, ...}, ...)
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/osmosis-labs/osmosis/v23/app/app.go:670 +0xbb
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(_, {{0xc0991e5100, 0x20, 0x20}, {{0xb, 0x0}, {0xc07b333402, 0x9}, 0xd7d30e, {0x37df29d0, ...}, ...}, ...})
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/cosmos/[email protected]/baseapp/abci.go:196 +0x838
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/cometbft/cometbft/abci/client.(*localClient).BeginBlockSync(_, {{0xc0991e5100, 0x20, 0x20}, {{0xb, 0x0}, {0xc07b333402, 0x9}, 0xd7d30e, {0x37df29d0, ...}, ...}, ...})
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/cometbft/[email protected]/abci/client/local_client.go:283 +0xfe
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/cometbft/cometbft/proxy.(*appConnConsensus).BeginBlockSync(_, {{0xc0991e5100, 0x20, 0x20}, {{0xb, 0x0}, {0xc07b333402, 0x9}, 0xd7d30e, {0x37df29d0, ...}, ...}, ...})
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/cometbft/[email protected]/proxy/app_conn.go:101 +0x195
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/cometbft/cometbft/state.execBlockOnProxyApp({0x3d4b210?, 0xc00161d4d0}, {0x3d60ee0, 0xc000cab920}, 0xc144fb9c20, {0x3d67318, 0xc00312bb30}, 0x3?)
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/cometbft/[email protected]/state/execution.go:375 +0x53d
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/cometbft/cometbft/state.(*BlockExecutor).applyBlock(_, {{{0xb, 0x0}, {0xc0012148a9, 0x7}}, {0xc001215190, 0x9}, 0x1, 0xd7d30d, {{0xc11e661860, ...}, ...}, ...}, ...)
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/cometbft/[email protected]/state/execution.go:209 +0x12e
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/cometbft/cometbft/state.(*BlockExecutor).ApplyVerifiedBlock(...)
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/cometbft/[email protected]/state/execution.go:186
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: github.com/cometbft/cometbft/blocksync.(*Reactor).poolRoutine(0xc002e1b340, 0x0)
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/cometbft/[email protected]/blocksync/reactor.go:423 +0xcba
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]: created by github.com/cometbft/cometbft/blocksync.(*Reactor).OnStart in goroutine 11
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:         github.com/cometbft/[email protected]/blocksync/reactor.go:129 +0x6e
Mar 05 17:27:11 multi-net-002 cosmovisor[1600058]:  module=server

Ideally, this wouldn't panic.

OTOH, pretty good that the chain kept moving :)

Osmosis Version

23.0.3-iavl-v1

How to reproduce?

During the epoch, the contract can't handle the stress and panics... 😏