hermes
hermes copied to clipboard
Published
20 hours ago •
informalsystems
informalsystems
Clearing packets apparently blocks Hermes from properly working
Summary of Bug
There might be multiple issues at once here, not sure whether it would be better to report these as a separate ones, but here we go. So, after something happened (not sure what, but still), Hermes started to think there's more than 20k packets pending:
monitoring@monitoring-logging ~ ❯ hermes query packet pending --chain osmosis-1 --channel channel-0 --port transfer
2024-06-30T20:15:58.243203Z INFO ThreadId(01) using default configuration from '/home/monitoring/.hermes/config.toml'
2024-06-30T20:15:58.244169Z INFO ThreadId(01) running Hermes v1.10.0+fc998de4
SUCCESS Summary of pending packets:
Packets pending on source chain (osmosis-1):
Unreceived packets:
3395732..=3396057
3396059..=3396063
3396065..=3396109
3396111..=3396210
3396212..=3396312
3396315..=3396458
3396460..=3396662
3396664..=3396712
3396715..=3396863
3396865..=3396902
3396904..=3397231
3397233..=3397251
3397255..=3397675
3397677..=3397709
3397711..=3398118
3398120..=3398932
3398934..=3399103
3399343..=3399367
3399369..=3399419
3399421..=3399425
3399427..=3399521
3399523..=3399680
3399682..=3399746
3399748..=3399833
3399836..=3399885
3399888..=3399937
3399939..=3400017
3400019..=3400039
3400041..=3400140
3400142..=3400191
3400193..=3400243
3400246..=3400402
3400405..=3400498
3400500..=3400531
3400533..=3400642
3400644..=3400670
3400672..=3400719
3400721..=3400881
3400883..=3401022
3401024..=3401503
3401505..=3401826
3401828..=3402110
3402112..=3402546
3402548..=3402568
3402571..=3402721
3402724..=3402774
3402776..=3402837
3402839..=3402876
3402878..=3403077
3403081..=3403283
3403287..=3403386
3403388..=3403438
3403440..=3403674
3403676..=3403681
3403683..=3403696
3403698..=3403749
3403751..=3403952
3403954..=3403974
3403976..=3404131
3404133..=3404147
3404149..=3404151
3404153..=3404156
3404158..=3404179
3404181..=3404296
3404298..=3404333
3404335..=3404360
3404362..=3404405
3404407..=3404418
3404420..=3404435
3404437..=3404448
3404450..=3404711
3404713..=3404737
3404739..=3404816
3404818..=3404820
3404822..=3404879
3404881..=3404928
3404930..=3404968
3404970..=3404998
3405000..=3405129
3405131..=3405195
3405197..=3405256
3405258..=3405347
3405349..=3405389
3405391..=3405397
3405399..=3405438
3405440..=3405577
3405579..=3405884
3405886..=3405904
3405906..=3405961
3405963..=3405966
3405968..=3406042
3406044..=3406183
3406185..=3406189
3406191..=3406280
3406282..=3406288
3406290..=3406299
3406301..=3406340
3406342..=3406351
3406353..=3406397
3406399..=3406402
3406404..=3406409
3406411..=3406493
3406495..=3406502
3406504..=3406507
3406509..=3406541
3406543..=3406630
3406632..=3406733
3406735..=3406791
3406793..=3406811
3406813..=3406933
3406935..=3406955
3406957..=3406958
3406960..=3407189
3407193..=3407241
3407243..=3407276
3407278..=3407293
3407295..=3407345
3407347..=3407396
3407399..=3407437
3407439..=3407549
3407551..=3407555
3407557..=3407601
3407603..=3407618
3407620..=3407662
3407664..=3407877
3407879..=3407934
3407936..=3408006
3408008..=3408057
3408059..=3408105
3408107..=3408136
3408138..=3408155
3408157..=3408159
3408161..=3408210
3408214..=3408263
3408265..=3408265
3408267..=3408280
3408282..=3408414
3408418..=3408466
3408468..=3408507
3408509..=3408517
3408519..=3408567
3408569..=3408638
3408640..=3408668
3408670..=3408769
3408771..=3408872
3408874..=3409026
3409028..=3409069
3409071..=3409153
3409155..=3409335
3409337..=3409385
3409387..=3409407
3409409..=3409427
3409429..=3409429
3409431..=3409575
3409577..=3409682
3409684..=3409692
3409694..=3409785
3409787..=3409826
3409828..=3410147
3410149..=3410160
3410162..=3410268
3410270..=3410283
3410285..=3410301
3410304..=3410565
3410567..=3410709
3410711..=3410717
3410719..=3410812
3410814..=3411214
3411216..=3411222
3411224..=3411431
3411433..=3411574
3411576..=3411630
3411633..=3411924
3411926..=3412294
3412296..=3412341
3412343..=3412599
3412601..=3412727
3412729..=3412837
3412839..=3413090
3413092..=3413154
3413156..=3413182
3413184..=3413206
3413208..=3413309
3413314..=3413367
3413369..=3413401
3413403..=3413633
3413635..=3413668
3413670..=3413918
3413920..=3414459
3414461..=3414473
3414475..=3414577
3414580..=3414628
3414630..=3414777
3414779..=3415100
3415102..=3415387
3415389..=3415458
3415460..=3415527
3415529..=3415685
3415687..=3415967
3415969..=3416011
3416013..=3416276
3416278..=3416377
3416380..=3416453
3416455..=3416711
3416713..=3416883
3416885..=3416910
3416912..=3416933
3416935..=3416985
3416987..=3417016
3417018..=3417087
3417089..=3417139
3417141..=3417181
3417183..=3417248
3417250..=3417293
3417295..=3417342
3417345..=3417390
3417392..=3417445
3417449..=3417497
3417499..=3417549
3417551..=3417582
3417584..=3417601
3417603..=3417613
3417615..=3417702
3417704..=3417754
3417756..=3417802
3417804..=3417804
3417808..=3417908
3417910..=3418338
3418340..=3418462
3418464..=3418515
3418517..=3418566
3418568..=3418944
3418946..=3419268
3419270..=3419340
3419342..=3419365
3419367..=3419576
3419578..=3419595
3419597..=3419616
3419618..=3419623
3419625..=3419675
3423615..=3423617
3423619..=3423620
3423622..=3423623
3423625..=3423627
3423629..=3423629
3423631..=3423634
3423636..=3423644
3423646..=3423646
3423648..=3423657
3423660..=3423664
3423666..=3423668
3423670..=3423692
3423695..=3423698
3423700..=3423708
3423712..=3423712
3423715..=3423719
3423721..=3423723
3423725..=3423729
3423731..=3423731
3423733..=3423741
3423743..=3423751
3423753..=3423759
3423761..=3423762
3423766..=3423771
3423773..=3423780
3423782..=3423787
3423789..=3423789
3423791..=3423794
3423796..=3423799
3423801..=3423811
3423813..=3423814
3423816..=3423820
3423822..=3423825
3423827..=3423829
3423831..=3423831
3423833..=3423834
3423836..=3423839
3423841..=3423843
3423845..=3423854
3423856..=3423867
3423869..=3423869
3423871..=3423884
3423886..=3423888
3423890..=3423893
3423896..=3423900
3423902..=3423906
3423908..=3423910
3423912..=3423917
3423919..=3423927
3423931..=3423931
3423933..=3423946
3423949..=3423950
3423952..=3423954
3423956..=3423968
3423970..=3423992
3423994..=3423995
3423998..=3424005
3462042..=3462043
Unreceived acks:
Packets pending on destination chain (cosmoshub-4):
Unreceived packets:
Unreceived acks:
3655048..=3655048
and it tries to clear these. Looking at the logs, this might take really long time (like more than 24h for sure), and during all this time, Hermes somehow isn't broadcasting any transactions (I can see it both from logs and from metrics). And apparently trying to clear the packets (both manually via hermes clear packets --chain osmosis-1 --channel channel-0 --port transfer and by waiting for Hermes main process packet clear routine to finish) doesn't work, as once it's done, Hermes runs the clean packets routine again, and everything is blocked again.
Basically there are 3 issues here:
- Hermes somehow thinks there are 20k packets pending from osmosis-1 -> cosmoshub-4
- When clearing packets, nothing happens, and the next time packets clearing routing is executed, it still shows the same 20k packets
- (the main one) Once the packet clear routine is being executed, Hermes isn't doing anything else.
I am not sure what can cause it and what was changed so it started behaving that way, but it seems wrong. I fixed it by disabling the check routine, but that seems like a workaround.
Here are Hermes logs:
Started Relayer.
2024-06-30T20:19:14.665887Z INFO ThreadId(01) using default configuration from '/home/monitoring/.hermes/config.toml'
2024-06-30T20:19:14.666410Z INFO ThreadId(01) running Hermes v1.10.0+fc998de4
2024-06-30T20:19:14.667780Z INFO ThreadId(01) rest: REST server disabled
2024-06-30T20:19:14.667832Z INFO ThreadId(19) telemetry service running, exposing metrics at http://127.0.0.1:3001/metrics
2024-06-30T20:19:14.688425Z WARN ThreadId(21) health_check{chain=cosmoshub-4}: found potential misconfiguration for chain 'cosmoshub-4'
2024-06-30T20:19:14.688441Z WARN ThreadId(21) health_check{chain=cosmoshub-4}: reason: semantic config validation failed for option `max_tx_size` for chain 'cosmoshub-4', reason: `max_tx_size` = 2000000 is greater than 90% of the consensus parameter `max_size` = 2000000
2024-06-30T20:19:14.688444Z WARN ThreadId(21) health_check{chain=cosmoshub-4}: some Hermes features may not work in this mode!
2024-06-30T20:19:14.688465Z WARN ThreadId(01) health_check{chain=cosmoshub-4}: chain is not healthy: semantic config validation failed for option `max_tx_size` for chain 'cosmoshub-4', reason: `max_tx_size` = 2000000 is greater than 90% of the consensus parameter `max_size` = 2000000
2024-06-30T20:19:15.768792Z WARN ThreadId(22) health_check{chain=osmosis-1}: chain 'osmosis-1' has no minimum gas price value configured for denomination 'uosmo'. This is usually a sign of misconfiguration, please check your chain and relayer configurations
2024-06-30T20:19:16.971506Z WARN ThreadId(22) health_check{chain=osmosis-1}: Will use fallback value for max_block_time: `30s`. Error: response error: Internal error: genesis response is large, please use the genesis_chunked API instead (code: -32603)
2024-06-30T20:19:16.971580Z INFO ThreadId(01) health_check{chain=osmosis-1}: chain is healthy
2024-06-30T20:19:16.972112Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}: scanning chain...
2024-06-30T20:19:16.972162Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}: chain uses an allow list (without wildcards), skipping scan for fast startup
2024-06-30T20:19:16.972167Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}: allowed ports/channels: transfer/channel-229, transfer/channel-141, transfer/channel-186
2024-06-30T20:19:16.972175Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}: querying allowed channels...
2024-06-30T20:19:16.972182Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-229}: querying channel...
2024-06-30T20:19:16.982881Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-229}: found connection and client connection=connection-395 client=07-tendermint-481
2024-06-30T20:19:16.982903Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-229}: querying client... client=07-tendermint-481
2024-06-30T20:19:16.984495Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-229}: found counterparty chain for client client=07-tendermint-481 counterparty_chain=bitsong-2b
2024-06-30T20:19:16.984524Z ERROR ThreadId(01) scan.chain{chain=cosmoshub-4}: failed to scan channel, reason: spawn: missing chain config for 'bitsong-2b' in configuration file channel=channel-229
2024-06-30T20:19:16.984536Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: querying channel...
2024-06-30T20:19:16.991021Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: found connection and client connection=connection-257 client=07-tendermint-259
2024-06-30T20:19:16.991039Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: querying client... client=07-tendermint-259
2024-06-30T20:19:16.992196Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: found counterparty chain for client client=07-tendermint-259 counterparty_chain=osmosis-1
2024-06-30T20:19:17.349801Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: found counterparty channel counterparty_channel=channel-0
2024-06-30T20:19:17.597171Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: found counterparty connection state counterparty_connection_state=OPEN
2024-06-30T20:19:17.597262Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-186}: querying channel...
2024-06-30T20:19:17.607225Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-186}: found connection and client connection=connection-341 client=07-tendermint-388
2024-06-30T20:19:17.607248Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-186}: querying client... client=07-tendermint-388
2024-06-30T20:19:17.608879Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-186}: found counterparty chain for client client=07-tendermint-388 counterparty_chain=sentinelhub-2
2024-06-30T20:19:17.608905Z ERROR ThreadId(01) scan.chain{chain=cosmoshub-4}: failed to scan channel, reason: spawn: missing chain config for 'sentinelhub-2' in configuration file channel=channel-186
2024-06-30T20:19:17.608924Z INFO ThreadId(01) scan.chain{chain=osmosis-1}: scanning chain...
2024-06-30T20:19:17.608976Z INFO ThreadId(01) scan.chain{chain=osmosis-1}: chain uses an allow list (without wildcards), skipping scan for fast startup
2024-06-30T20:19:17.608980Z INFO ThreadId(01) scan.chain{chain=osmosis-1}: allowed ports/channels: transfer/channel-73, transfer/channel-0, transfer/channel-412, transfer/channel-2, wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn/channel-78439
2024-06-30T20:19:17.608989Z INFO ThreadId(01) scan.chain{chain=osmosis-1}: querying allowed channels...
2024-06-30T20:19:17.608994Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-73}: querying channel...
2024-06-30T20:19:18.215471Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-73}: found connection and client connection=connection-1216 client=07-tendermint-1551
2024-06-30T20:19:18.215497Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-73}: querying client... client=07-tendermint-1551
2024-06-30T20:19:18.334420Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-73}: found counterparty chain for client client=07-tendermint-1551 counterparty_chain=bitsong-2b
2024-06-30T20:19:18.334460Z ERROR ThreadId(01) scan.chain{chain=osmosis-1}: failed to scan channel, reason: spawn: missing chain config for 'bitsong-2b' in configuration file channel=channel-73
2024-06-30T20:19:18.334472Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: querying channel...
2024-06-30T20:19:18.691059Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: found connection and client connection=connection-1 client=07-tendermint-1
2024-06-30T20:19:18.691085Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: querying client... client=07-tendermint-1
2024-06-30T20:19:18.809703Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: found counterparty chain for client client=07-tendermint-1 counterparty_chain=cosmoshub-4
2024-06-30T20:19:18.816134Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: found counterparty channel counterparty_channel=channel-141
2024-06-30T20:19:18.816177Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: found counterparty connection state counterparty_connection_state=OPEN
2024-06-30T20:19:18.816235Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-412}: querying channel...
2024-06-30T20:19:19.418229Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-412}: found connection and client connection=connection-1762 client=07-tendermint-2273
2024-06-30T20:19:19.418258Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-412}: querying client... client=07-tendermint-2273
2024-06-30T20:19:19.536807Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-412}: found counterparty chain for client client=07-tendermint-2273 counterparty_chain=jackal-1
2024-06-30T20:19:19.536851Z ERROR ThreadId(01) scan.chain{chain=osmosis-1}: failed to scan channel, reason: spawn: missing chain config for 'jackal-1' in configuration file channel=channel-412
2024-06-30T20:19:19.536864Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-2}: querying channel...
2024-06-30T20:19:20.138088Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-2}: found connection and client connection=connection-3 client=07-tendermint-2
2024-06-30T20:19:20.138116Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-2}: querying client... client=07-tendermint-2
2024-06-30T20:19:20.257000Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-2}: found counterparty chain for client client=07-tendermint-2 counterparty_chain=sentinelhub-2
2024-06-30T20:19:20.257038Z ERROR ThreadId(01) scan.chain{chain=osmosis-1}: failed to scan channel, reason: spawn: missing chain config for 'sentinelhub-2' in configuration file channel=channel-2
2024-06-30T20:19:20.257053Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn channel=channel-78439}: querying channel...
2024-06-30T20:19:20.615793Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn channel=channel-78439}: found connection and client connection=connection-1216 client=07-tendermint-1551
2024-06-30T20:19:20.615821Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn channel=channel-78439}: querying client... client=07-tendermint-1551
2024-06-30T20:19:20.843420Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn channel=channel-78439}: found counterparty chain for client client=07-tendermint-1551 counterparty_chain=bitsong-2b
2024-06-30T20:19:20.843458Z ERROR ThreadId(01) scan.chain{chain=osmosis-1}: failed to scan channel, reason: spawn: missing chain config for 'bitsong-2b' in configuration file channel=channel-78439
2024-06-30T20:19:20.843476Z INFO ThreadId(01) scanned chains:
2024-06-30T20:19:20.843480Z INFO ThreadId(01) # Chain: cosmoshub-4
- Client: 07-tendermint-259
* Connection: connection-257
| State: OPEN
| Counterparty state: OPEN
+ Channel: channel-141
| Port: transfer
| State: OPEN
| Counterparty: channel-0
# Chain: osmosis-1
- Client: 07-tendermint-1
* Connection: connection-1
| State: OPEN
| Counterparty state: OPEN
+ Channel: channel-0
| Port: transfer
| State: OPEN
| Counterparty: channel-141
2024-06-30T20:19:20.843516Z INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}: connection is OPEN, state on destination chain is OPEN chain=cosmoshub-4 connection=connection-257 counterparty_chain=osmosis-1
2024-06-30T20:19:20.843523Z INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}: connection is already open, not spawning Connection worker chain=cosmoshub-4 connection=connection-257
2024-06-30T20:19:20.843531Z INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}: no connection workers were spawn chain=cosmoshub-4 connection=connection-257
2024-06-30T20:19:20.843539Z INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}: channel is OPEN, state on destination chain is OPEN chain=cosmoshub-4 counterparty_chain=osmosis-1 channel=channel-141
2024-06-30T20:19:21.116508Z INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}: spawned client worker: client::osmosis-1->cosmoshub-4:07-tendermint-259
2024-06-30T20:19:24.629223Z INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}: done spawning channel workers chain=cosmoshub-4 channel=channel-141
2024-06-30T20:19:24.629334Z INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}: spawning Wallet worker: wallet::cosmoshub-4
2024-06-30T20:19:24.629371Z INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}: connection is OPEN, state on destination chain is OPEN chain=osmosis-1 connection=connection-1 counterparty_chain=cosmoshub-4
2024-06-30T20:19:24.629379Z INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}: connection is already open, not spawning Connection worker chain=osmosis-1 connection=connection-1
2024-06-30T20:19:24.629386Z INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}: no connection workers were spawn chain=osmosis-1 connection=connection-1
2024-06-30T20:19:24.629393Z INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}: channel is OPEN, state on destination chain is OPEN chain=osmosis-1 counterparty_chain=cosmoshub-4 channel=channel-0
2024-06-30T20:19:24.871166Z INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}: spawned client worker: client::cosmoshub-4->osmosis-1:07-tendermint-1
2024-06-30T20:19:28.461923Z INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}: spawned packet worker: packet::channel-0/transfer:osmosis-1->cosmoshub-4
2024-06-30T20:19:28.461950Z INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}: done spawning channel workers chain=osmosis-1 channel=channel-0
2024-06-30T20:19:28.461998Z INFO ThreadId(01) spawn:chain{chain=osmosis-1}: spawning Wallet worker: wallet::osmosis-1
2024-06-30T20:19:29.779724Z INFO ThreadId(01) Hermes has started
2024-06-30T20:19:32.664232Z INFO ThreadId(26) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}: clearing packets
2024-06-30T20:20:06.201132Z INFO ThreadId(26) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 17369387 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 17369386 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-17369386}: pulled packet data for 0 events out of 50 sequences: 3395732..=3395781; events.total=23757 events.left=23707
2024-06-30T20:20:37.426075Z INFO ThreadId(26) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 17369387 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 17369386 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-17369386}: pulled packet data for 0 events out of 50 sequences: 3395782..=3395831; events.total=23757 events.left=23657
2024-06-30T20:21:08.440887Z INFO ThreadId(26) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 17369387 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 17369386 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-17369386}: pulled packet data for 0 events out of 50 sequences: 3395832..=3395881; events.total=23757 events.left=23607
2024-06-30T20:21:39.215104Z INFO ThreadId(26) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 17369387 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 17369386 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-17369386}: pulled packet data for 0 events out of 50 sequences: 3395882..=3395931; events.total=23757 events.left=23557
2024-06-30T20:22:10.734854Z INFO ThreadId(26) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 17369387 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 17369386 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-17369386}: pulled packet data for 0 events out of 50 sequences: 3395932..=3395981; events.total=23757 events.left=23507
# and many many like the previous one
And here's the minimal config I was able to reproduce it with:
[global]
log_level = "info"
[mode.clients]
enabled = true
refresh = true
misbehaviour = false
[mode.connections]
enabled = false
[mode.channels]
enabled = false
[mode.packets]
enabled = true
# Commenting out these 2 lines and uncommenting the next 2 lines after that one fixes it.
clear_interval = 100
clear_on_start = true
# clear_interval = 0
# clear_on_start = false
tx_confirmation = true
auto_register_counterparty_payee = false
ics20_max_memo_size = { enabled = true, size = "32KiB" }
ics20_max_receiver_size = { enabled = true, size = "2KiB" }
[rest]
enabled = false
host = "127.0.0.1"
port = 3000
[telemetry]
enabled = true
host = "127.0.0.1"
port = 3001
[telemetry.buckets.latency_submitted]
start = 500
end = 20000
buckets = 10
[telemetry.buckets.latency_confirmed]
start = 1000
end = 30000
buckets = 10
[[chains]]
id = "cosmoshub-4"
type = "CosmosSdk"
rpc_addr = "https://rpc.cosmos.quokkastake.io"
grpc_addr = "http://rpc.cosmos.quokkastake.io:9090"
rpc_timeout = "20s"
trusted_node = false
account_prefix = "cosmos"
key_name = "relayer"
key_store_type = "Test"
store_prefix = "ibc"
default_gas = 800000
max_gas = 3500000
gas_multiplier = 1.5
max_msg_num = 30
max_tx_size = 2000000
max_grpc_decoding_size = 33554432
clock_drift = "5s"
max_block_time = "30s"
ccv_consumer_chain = false
memo_prefix = "Relayed with ❤️ by 🐹 Quokka Stake"
sequential_batch_tx = false
event_source = { mode = "push", url = "wss://rpc.cosmos.quokkastake.io/websocket", batch_delay = "1000ms" }
trust_threshold = { numerator = "1", denominator = "3" }
gas_price = { price = 0.005, denom = "uatom" }
address_type = { derivation = "cosmos" }
[chains.packet_filter]
policy = "allow"
list = [
["transfer", "channel-229"],
["transfer", "channel-141"],
["transfer", "channel-186"],
]
[[chains]]
id = "osmosis-1"
type = "CosmosSdk"
rpc_addr = "https://rpc-osmosis.sentinelgrowthdao.com"
grpc_addr = "http://rpc-osmosis.sentinelgrowthdao.com:9090"
rpc_timeout = "600s"
trusted_node = false
account_prefix = "osmo"
key_name = "relayer"
key_store_type = "Test"
store_prefix = "ibc"
default_gas = 1000000
max_gas = 120000000
gas_multiplier = 1.4
max_msg_num = 30
max_tx_size = 1800000
max_grpc_decoding_size = 33554432
clock_drift = "5s"
max_block_time = "30s"
ccv_consumer_chain = false
memo_prefix = "Relayed with ❤️ by 🐹 Quokka Stake"
sequential_batch_tx = false
trust_threshold = { numerator = "1", denominator = "3" }
gas_price = { price = 0.01, denom = 'uosmo' }
event_source = { mode = "push", url = "wss://rpc-osmosis.sentinelgrowthdao.com/websocket", batch_delay = "500ms" }
address_type = { derivation = "cosmos" }
[chains.packet_filter]
policy = "allow"
list = [
["transfer", "channel-73"],
["transfer", "channel-0"],
["transfer", "channel-412"],
["transfer", "channel-2"],
["wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn", "channel-78439"] # polytone from bitsong-2b
]
[tracing_server]
enabled = false
port = 5555
Version
hermes 1.10.0+fc998de4
Steps to Reproduce
See above for the minimal config example and for logs.
Acceptance Criteria
For Admin Use
- [ ] Not duplicate issue
- [ ] Appropriate labels applied
- [ ] Appropriate milestone (priority) applied
- [ ] Appropriate contributors tagged
- [ ] Contributor assigned/self-assigned
Might be related: https://github.com/informalsystems/hermes/issues/2612
A first issue I have noticed with the RPC endpoint is that it doesn't seem to persist ABCI responses. For example, if I try to query https://rpc-osmosis.sentinelgrowthdao.com/block_results?height=17198597 I will get:
{"jsonrpc":"2.0","id":-1,"error":{"code":-32603,"message":"Internal error","data":"node is not persisting abci responses"}}
This is usually due to the CometBFT configuration discard_abci_responses being set to true. Could you verify this configuration?
And have you tried using a Node with more historical data? The one being used doesn't seem to have the packet data anymore: pulled packet data for 0 events out of 50 sequences.
@ljoss17 most likely, will check. I do not have Osmosis archive node unfortunately, so I cannot check it with older heights, but I will check the discard_abci_responses a bit later.
The pulled packet data for 0 events out of 50 sequences part also a bit counter-unintuitive; IMO it should be a warning (like, reporting something like could not pull X/50 events if X > 0) and it ideally should report the reason what was the issue when pulling events, do you think it's possible?
It is a valid point, this log message can be confusing, thanks for the feedback! I will look into this and see what information I can extract from the query and improve the log message.
@ljoss17 did you also check the point about clear packets routine thing blocking the rest? This seems to be the biggest issue here, for me at least.
I will look into that right now. Would you be able to share logs at trace level with the instance running for a few minutes while the clearing is triggered? Just a heads-up, this will generate a lot of logs, so you might need to store the output to a file.
@ljoss17 here you go, let me know if you need more.
2024-07-01T11:54:03.805896Z INFO ThreadId(01) using default configuration from '/home/monitoring/.hermes/config.toml'
2024-07-01T11:54:03.806405Z INFO ThreadId(01) running Hermes v1.10.0+fc998de4
2024-07-01T11:54:03.807674Z INFO ThreadId(01) rest: REST server disabled
2024-07-01T11:54:03.807789Z INFO ThreadId(19) telemetry service running, exposing metrics at http://127.0.0.1:3001/metrics
2024-07-01T11:54:03.815481Z TRACE ThreadId(01) health_check{chain=cosmoshub-4}: spawned chain runtime chain=cosmoshub-4
2024-07-01T11:54:03.829073Z TRACE ThreadId(21) health_check{chain=cosmoshub-4}: parsed version specification application=gaiad version=v17.2.0 git_commit=17f2ba0b90d1d2884f7b67518ec08dfd37f001a7 sdk_version=Some(Version { major: 0, minor: 47, patch: 11 }) ibc_go_status=Some(Version { major: 7, minor: 4, patch: 0 }) consensus_version=Some(Comet(Version { major: 0, minor: 37, patch: 6 }))
2024-07-01T11:54:03.829103Z DEBUG ThreadId(21) health_check{chain=cosmoshub-4}: running diagnostic on version specs: Cosmos SDK 0.47.11, IBC-Go 7.4.0, CometBFT 0.37.6
2024-07-01T11:54:03.836498Z WARN ThreadId(21) health_check{chain=cosmoshub-4}: found potential misconfiguration for chain 'cosmoshub-4'
2024-07-01T11:54:03.836513Z WARN ThreadId(21) health_check{chain=cosmoshub-4}: reason: semantic config validation failed for option `max_tx_size` for chain 'cosmoshub-4', reason: `max_tx_size` = 2000000 is greater than 90% of the consensus parameter `max_size` = 2000000
2024-07-01T11:54:03.836516Z WARN ThreadId(21) health_check{chain=cosmoshub-4}: some Hermes features may not work in this mode!
2024-07-01T11:54:03.836541Z WARN ThreadId(01) health_check{chain=cosmoshub-4}: chain is not healthy: semantic config validation failed for option `max_tx_size` for chain 'cosmoshub-4', reason: `max_tx_size` = 2000000 is greater than 90% of the consensus parameter `max_size` = 2000000
2024-07-01T11:54:04.207422Z TRACE ThreadId(01) health_check{chain=osmosis-1}: spawned chain runtime chain=osmosis-1
2024-07-01T11:54:04.948556Z WARN ThreadId(22) health_check{chain=osmosis-1}: chain 'osmosis-1' has no minimum gas price value configured for denomination 'uosmo'. This is usually a sign of misconfiguration, please check your chain and relayer configurations
2024-07-01T11:54:05.311784Z TRACE ThreadId(22) health_check{chain=osmosis-1}: parsed version specification application=osmosisd version=25.1.3 git_commit=dd51aa80ce509c7577238580feda8246bfbe7b82 sdk_version=Some(Version { major: 0, minor: 47, patch: 8 }) ibc_go_status=Some(Version { major: 7, minor: 4, patch: 1 }) consensus_version=Some(Comet(Version { major: 0, minor: 37, patch: 4 }))
2024-07-01T11:54:05.311828Z DEBUG ThreadId(22) health_check{chain=osmosis-1}: running diagnostic on version specs: Cosmos SDK 0.47.8, IBC-Go 7.4.1, CometBFT 0.37.4
2024-07-01T11:54:06.174247Z WARN ThreadId(22) health_check{chain=osmosis-1}: Will use fallback value for max_block_time: `30s`. Error: response error: Internal error: genesis response is large, please use the genesis_chunked API instead (code: -32603)
2024-07-01T11:54:06.174286Z INFO ThreadId(01) health_check{chain=osmosis-1}: chain is healthy
2024-07-01T11:54:06.174743Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}: scanning chain...
2024-07-01T11:54:06.174796Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}: chain uses an allow list (without wildcards), skipping scan for fast startup
2024-07-01T11:54:06.174800Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}: allowed ports/channels: transfer/channel-229, transfer/channel-141, transfer/channel-186
2024-07-01T11:54:06.174815Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}: querying allowed channels...
2024-07-01T11:54:06.174822Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-229}: querying channel...
2024-07-01T11:54:06.184413Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-229}: found connection and client connection=connection-395 client=07-tendermint-481
2024-07-01T11:54:06.184440Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-229}: querying client... client=07-tendermint-481
2024-07-01T11:54:06.186133Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-229}: found counterparty chain for client client=07-tendermint-481 counterparty_chain=bitsong-2b
2024-07-01T11:54:06.186165Z ERROR ThreadId(01) scan.chain{chain=cosmoshub-4}: failed to scan channel, reason: spawn: missing chain config for 'bitsong-2b' in configuration file channel=channel-229
2024-07-01T11:54:06.186178Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: querying channel...
2024-07-01T11:54:06.193753Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: found connection and client connection=connection-257 client=07-tendermint-259
2024-07-01T11:54:06.193775Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: querying client... client=07-tendermint-259
2024-07-01T11:54:06.195314Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: found counterparty chain for client client=07-tendermint-259 counterparty_chain=osmosis-1
2024-07-01T11:54:06.567450Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: found counterparty channel counterparty_channel=channel-0
2024-07-01T11:54:06.815949Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: found counterparty connection state counterparty_connection_state=OPEN
2024-07-01T11:54:06.816054Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-186}: querying channel...
2024-07-01T11:54:06.825456Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-186}: found connection and client connection=connection-341 client=07-tendermint-388
2024-07-01T11:54:06.825480Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-186}: querying client... client=07-tendermint-388
2024-07-01T11:54:06.827001Z INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-186}: found counterparty chain for client client=07-tendermint-388 counterparty_chain=sentinelhub-2
2024-07-01T11:54:06.827029Z ERROR ThreadId(01) scan.chain{chain=cosmoshub-4}: failed to scan channel, reason: spawn: missing chain config for 'sentinelhub-2' in configuration file channel=channel-186
2024-07-01T11:54:06.827045Z INFO ThreadId(01) scan.chain{chain=osmosis-1}: scanning chain...
2024-07-01T11:54:06.827086Z INFO ThreadId(01) scan.chain{chain=osmosis-1}: chain uses an allow list (without wildcards), skipping scan for fast startup
2024-07-01T11:54:06.827088Z INFO ThreadId(01) scan.chain{chain=osmosis-1}: allowed ports/channels: transfer/channel-73, transfer/channel-0, transfer/channel-412, transfer/channel-2, wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn/channel-78439
2024-07-01T11:54:06.827095Z INFO ThreadId(01) scan.chain{chain=osmosis-1}: querying allowed channels...
2024-07-01T11:54:06.827098Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-73}: querying channel...
2024-07-01T11:54:07.445781Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-73}: found connection and client connection=connection-1216 client=07-tendermint-1551
2024-07-01T11:54:07.445816Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-73}: querying client... client=07-tendermint-1551
2024-07-01T11:54:07.568820Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-73}: found counterparty chain for client client=07-tendermint-1551 counterparty_chain=bitsong-2b
2024-07-01T11:54:07.568861Z ERROR ThreadId(01) scan.chain{chain=osmosis-1}: failed to scan channel, reason: spawn: missing chain config for 'bitsong-2b' in configuration file channel=channel-73
2024-07-01T11:54:07.568873Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: querying channel...
2024-07-01T11:54:07.939226Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: found connection and client connection=connection-1 client=07-tendermint-1
2024-07-01T11:54:07.939255Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: querying client... client=07-tendermint-1
2024-07-01T11:54:08.062658Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: found counterparty chain for client client=07-tendermint-1 counterparty_chain=cosmoshub-4
2024-07-01T11:54:08.070205Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: found counterparty channel counterparty_channel=channel-141
2024-07-01T11:54:08.070256Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: found counterparty connection state counterparty_connection_state=OPEN
2024-07-01T11:54:08.070315Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-412}: querying channel...
2024-07-01T11:54:08.705140Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-412}: found connection and client connection=connection-1762 client=07-tendermint-2273
2024-07-01T11:54:08.705169Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-412}: querying client... client=07-tendermint-2273
2024-07-01T11:54:08.829093Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-412}: found counterparty chain for client client=07-tendermint-2273 counterparty_chain=jackal-1
2024-07-01T11:54:08.829132Z ERROR ThreadId(01) scan.chain{chain=osmosis-1}: failed to scan channel, reason: spawn: missing chain config for 'jackal-1' in configuration file channel=channel-412
2024-07-01T11:54:08.829143Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-2}: querying channel...
2024-07-01T11:54:09.453813Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-2}: found connection and client connection=connection-3 client=07-tendermint-2
2024-07-01T11:54:09.453834Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-2}: querying client... client=07-tendermint-2
2024-07-01T11:54:09.576891Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-2}: found counterparty chain for client client=07-tendermint-2 counterparty_chain=sentinelhub-2
2024-07-01T11:54:09.576935Z ERROR ThreadId(01) scan.chain{chain=osmosis-1}: failed to scan channel, reason: spawn: missing chain config for 'sentinelhub-2' in configuration file channel=channel-2
2024-07-01T11:54:09.576947Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn channel=channel-78439}: querying channel...
2024-07-01T11:54:09.948716Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn channel=channel-78439}: found connection and client connection=connection-1216 client=07-tendermint-1551
2024-07-01T11:54:09.948745Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn channel=channel-78439}: querying client... client=07-tendermint-1551
2024-07-01T11:54:10.071991Z INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn channel=channel-78439}: found counterparty chain for client client=07-tendermint-1551 counterparty_chain=bitsong-2b
2024-07-01T11:54:10.072032Z ERROR ThreadId(01) scan.chain{chain=osmosis-1}: failed to scan channel, reason: spawn: missing chain config for 'bitsong-2b' in configuration file channel=channel-78439
2024-07-01T11:54:10.072050Z INFO ThreadId(01) scanned chains:
2024-07-01T11:54:10.072052Z INFO ThreadId(01) # Chain: cosmoshub-4
- Client: 07-tendermint-259
* Connection: connection-257
| State: OPEN
| Counterparty state: OPEN
+ Channel: channel-141
| Port: transfer
| State: OPEN
| Counterparty: channel-0
# Chain: osmosis-1
- Client: 07-tendermint-1
* Connection: connection-1
| State: OPEN
| Counterparty state: OPEN
+ Channel: channel-0
| Port: transfer
| State: OPEN
| Counterparty: channel-141
2024-07-01T11:54:10.072100Z INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}: connection is OPEN, state on destination chain is OPEN chain=cosmoshub-4 connection=connection-257 counterparty_chain=osmosis-1
2024-07-01T11:54:10.072105Z INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}: connection is already open, not spawning Connection worker chain=cosmoshub-4 connection=connection-257
2024-07-01T11:54:10.072112Z INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}: no connection workers were spawn chain=cosmoshub-4 connection=connection-257
2024-07-01T11:54:10.072118Z INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}: channel is OPEN, state on destination chain is OPEN chain=cosmoshub-4 counterparty_chain=osmosis-1 channel=channel-141
2024-07-01T11:54:10.321022Z DEBUG ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.client.refresh{client=07-tendermint-259 src_chain=osmosis-1 dst_chain=cosmoshub-4}: spawning task
2024-07-01T11:54:10.321130Z INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}: spawned client worker: client::osmosis-1->cosmoshub-4:07-tendermint-259
2024-07-01T11:54:12.695528Z INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}: done spawning channel workers chain=cosmoshub-4 channel=channel-141
2024-07-01T11:54:12.695587Z DEBUG ThreadId(01) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: spawning task
2024-07-01T11:54:12.695651Z INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}: spawning Wallet worker: wallet::cosmoshub-4
2024-07-01T11:54:12.695683Z INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}: connection is OPEN, state on destination chain is OPEN chain=osmosis-1 connection=connection-1 counterparty_chain=cosmoshub-4
2024-07-01T11:54:12.695688Z INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}: connection is already open, not spawning Connection worker chain=osmosis-1 connection=connection-1
2024-07-01T11:54:12.695692Z INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}: no connection workers were spawn chain=osmosis-1 connection=connection-1
2024-07-01T11:54:12.695698Z INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}: channel is OPEN, state on destination chain is OPEN chain=osmosis-1 counterparty_chain=cosmoshub-4 channel=channel-0
2024-07-01T11:54:12.698097Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:54:12.944423Z DEBUG ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.client.refresh{client=07-tendermint-1 src_chain=cosmoshub-4 dst_chain=osmosis-1}: spawning task
2024-07-01T11:54:12.944513Z INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}: spawned client worker: client::cosmoshub-4->osmosis-1:07-tendermint-1
2024-07-01T11:54:15.154861Z DEBUG ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}: spawning task
2024-07-01T11:54:15.154951Z DEBUG ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}: spawning task
2024-07-01T11:54:15.155002Z INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}: spawned packet worker: packet::channel-0/transfer:osmosis-1->cosmoshub-4
2024-07-01T11:54:15.155015Z INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}: done spawning channel workers chain=osmosis-1 channel=channel-0
2024-07-01T11:54:15.155032Z DEBUG ThreadId(01) spawn:chain{chain=osmosis-1}:wallet{chain=osmosis-1}: spawning task
2024-07-01T11:54:15.155065Z INFO ThreadId(01) spawn:chain{chain=osmosis-1}: spawning Wallet worker: wallet::osmosis-1
2024-07-01T11:54:15.160407Z TRACE ThreadId(21) supervisor.init_subscriptions:event_source.init_subscriptions{chain=cosmoshub-4}: subscribing to query: tm.event = 'NewBlock'
2024-07-01T11:54:15.160820Z TRACE ThreadId(21) supervisor.init_subscriptions:event_source.init_subscriptions{chain=cosmoshub-4}: subscribing to query: message.module = 'ibc_client'
2024-07-01T11:54:15.161208Z TRACE ThreadId(21) supervisor.init_subscriptions:event_source.init_subscriptions{chain=cosmoshub-4}: subscribing to query: message.module = 'ibc_connection'
2024-07-01T11:54:15.161460Z TRACE ThreadId(21) supervisor.init_subscriptions:event_source.init_subscriptions{chain=cosmoshub-4}: subscribing to query: message.module = 'ibc_channel'
2024-07-01T11:54:15.161700Z TRACE ThreadId(21) supervisor.init_subscriptions:event_source.init_subscriptions{chain=cosmoshub-4}: subscribing to query: message.module = 'interchainquery'
2024-07-01T11:54:15.161931Z TRACE ThreadId(21) supervisor.init_subscriptions:event_source.init_subscriptions{chain=cosmoshub-4}: subscribed to all queries
2024-07-01T11:54:15.162066Z DEBUG ThreadId(30) event_source.websocket{chain=cosmoshub-4}: collecting events
2024-07-01T11:54:15.402123Z TRACE ThreadId(28) spawn:chain{chain=osmosis-1}:wallet{chain=osmosis-1}: wallet balance amount=47405742 denom=uosmo account=osmo17ndx5qfku28ymxgmq6zq4a6d02dvpfjjul0hyh
2024-07-01T11:54:15.893597Z TRACE ThreadId(22) supervisor.init_subscriptions:event_source.init_subscriptions{chain=osmosis-1}: subscribing to query: tm.event = 'NewBlock'
2024-07-01T11:54:16.015357Z TRACE ThreadId(22) supervisor.init_subscriptions:event_source.init_subscriptions{chain=osmosis-1}: subscribing to query: message.module = 'ibc_client'
2024-07-01T11:54:16.137118Z TRACE ThreadId(22) supervisor.init_subscriptions:event_source.init_subscriptions{chain=osmosis-1}: subscribing to query: message.module = 'ibc_connection'
2024-07-01T11:54:16.258815Z TRACE ThreadId(22) supervisor.init_subscriptions:event_source.init_subscriptions{chain=osmosis-1}: subscribing to query: message.module = 'ibc_channel'
2024-07-01T11:54:16.380717Z TRACE ThreadId(22) supervisor.init_subscriptions:event_source.init_subscriptions{chain=osmosis-1}: subscribing to query: message.module = 'interchainquery'
2024-07-01T11:54:16.502562Z TRACE ThreadId(22) supervisor.init_subscriptions:event_source.init_subscriptions{chain=osmosis-1}: subscribed to all queries
2024-07-01T11:54:16.502714Z DEBUG ThreadId(31) event_source.websocket{chain=osmosis-1}: collecting events
2024-07-01T11:54:16.502834Z DEBUG ThreadId(01) worker.batch{chain=cosmoshub-4}: spawning task
2024-07-01T11:54:16.502926Z DEBUG ThreadId(01) worker.batch{chain=osmosis-1}: spawning task
2024-07-01T11:54:16.502952Z DEBUG ThreadId(01) worker.cmd: spawning task
2024-07-01T11:54:16.502994Z DEBUG ThreadId(01) cleanup_worker: spawning task
2024-07-01T11:54:16.503094Z INFO ThreadId(01) Hermes has started
2024-07-01T11:54:16.889123Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:17.389731Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:54:17.389773Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395791
2024-07-01T11:54:17.556227Z INFO ThreadId(26) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}: clearing packets
2024-07-01T11:54:17.702882Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:54:19.398574Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: tm.event = 'NewBlock' chain=cosmoshub-4
2024-07-01T11:54:19.799829Z DEBUG ThreadId(26) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 17395791 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 17395790 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-17395790}: sequence numbers of unreceived packets to send to the destination chain out of the ones with commitments on the source chain dst_chain=cosmoshub-4 src_chain=osmosis-1 total=23757 sequences=3395732..=3396057, 3396059..=3396063, 3396065..=3396109, 3396111..=3396210, 3396212..=3396312, 3396315..=3396458, 3396460..=3396662, 3396664..=3396712, 3396715..=3396863, 3396865..=3396902, 3396904..=3397231, 3397233..=3397251, 3397255..=3397675, 3397677..=3397709, 3397711..=3398118, 3398120..=3398932, 3398934..=3399103, 3399343..=3399367, 3399369..=3399419, 3399421..=3399425, 3399427..=3399521, 3399523..=3399680, 3399682..=3399746, 3399748..=3399833, 3399836..=3399885, 3399888..=3399937, 3399939..=3400017, 3400019..=3400039, 3400041..=3400140, 3400142..=3400191, 3400193..=3400243, 3400246..=3400402, 3400405..=3400498, 3400500..=3400531, 3400533..=3400642, 3400644..=3400670, 3400672..=3400719, 3400721..=3400881, 3400883..=3401022, 3401024..=3401503, 3401505..=3401826, 3401828..=3402110, 3402112..=3402546, 3402548..=3402568, 3402571..=3402721, 3402724..=3402774, 3402776..=3402837, 3402839..=3402876, 3402878..=3403077, 3403081..=3403283, 3403287..=3403386, 3403388..=3403438, 3403440..=3403674, 3403676..=3403681, 3403683..=3403696, 3403698..=3403749, 3403751..=3403952, 3403954..=3403974, 3403976..=3404131, 3404133..=3404147, 3404149..=3404151, 3404153..=3404156, 3404158..=3404179, 3404181..=3404296, 3404298..=3404333, 3404335..=3404360, 3404362..=3404405, 3404407..=3404418, 3404420..=3404435, 3404437..=3404448, 3404450..=3404711, 3404713..=3404737, 3404739..=3404816, 3404818..=3404820, 3404822..=3404879, 3404881..=3404928, 3404930..=3404968, 3404970..=3404998, 3405000..=3405129, 3405131..=3405195, 3405197..=3405256, 3405258..=3405347, 3405349..=3405389, 3405391..=3405397, 3405399..=3405438, 3405440..=3405577, 3405579..=3405884, 3405886..=3405904, 3405906..=3405961, 3405963..=3405966, 3405968..=3406042, 3406044..=3406183, 3406185..=3406189, 3406191..=3406280, 3406282..=3406288, 3406290..=3406299, 3406301..=3406340, 3406342..=3406351, 3406353..=3406397, 3406399..=3406402, 3406404..=3406409, 3406411..=3406493, 3406495..=3406502, 3406504..=3406507, 3406509..=3406541, 3406543..=3406630, 3406632..=3406733, 3406735..=3406791, 3406793..=3406811, 3406813..=3406933, 3406935..=3406955, 3406957..=3406958, 3406960..=3407189, 3407193..=3407241, 3407243..=3407276, 3407278..=3407293, 3407295..=3407345, 3407347..=3407396, 3407399..=3407437, 3407439..=3407549, 3407551..=3407555, 3407557..=3407601, 3407603..=3407618, 3407620..=3407662, 3407664..=3407877, 3407879..=3407934, 3407936..=3408006, 3408008..=3408057, 3408059..=3408105, 3408107..=3408136, 3408138..=3408155, 3408157..=3408159, 3408161..=3408210, 3408214..=3408263, 3408265..=3408265, 3408267..=3408280, 3408282..=3408414, 3408418..=3408466, 3408468..=3408507, 3408509..=3408517, 3408519..=3408567, 3408569..=3408638, 3408640..=3408668, 3408670..=3408769, 3408771..=3408872, 3408874..=3409026, 3409028..=3409069, 3409071..=3409153, 3409155..=3409335, 3409337..=3409385, 3409387..=3409407, 3409409..=3409427, 3409429..=3409429, 3409431..=3409575, 3409577..=3409682, 3409684..=3409692, 3409694..=3409785, 3409787..=3409826, 3409828..=3410147, 3410149..=3410160, 3410162..=3410268, 3410270..=3410283, 3410285..=3410301, 3410304..=3410565, 3410567..=3410709, 3410711..=3410717, 3410719..=3410812, 3410814..=3411214, 3411216..=3411222, 3411224..=3411431, 3411433..=3411574, 3411576..=3411630, 3411633..=3411924, 3411926..=3412294, 3412296..=3412341, 3412343..=3412599, 3412601..=3412727, 3412729..=3412837, 3412839..=3413090, 3413092..=3413154, 3413156..=3413182, 3413184..=3413206, 3413208..=3413309, 3413314..=3413367, 3413369..=3413401, 3413403..=3413633, 3413635..=3413668, 3413670..=3413918, 3413920..=3414459, 3414461..=3414473, 3414475..=3414577, 3414580..=3414628, 3414630..=3414777, 3414779..=3415100, 3415102..=3415387, 3415389..=3415458, 3415460..=3415527, 3415529..=3415685, 3415687..=3415967, 3415969..=3416011, 3416013..=3416276, 3416278..=3416377, 3416380..=3416453, 3416455..=3416711, 3416713..=3416883, 3416885..=3416910, 3416912..=3416933, 3416935..=3416985, 3416987..=3417016, 3417018..=3417087, 3417089..=3417139, 3417141..=3417181, 3417183..=3417248, 3417250..=3417293, 3417295..=3417342, 3417345..=3417390, 3417392..=3417445, 3417449..=3417497, 3417499..=3417549, 3417551..=3417582, 3417584..=3417601, 3417603..=3417613, 3417615..=3417702, 3417704..=3417754, 3417756..=3417802, 3417804..=3417804, 3417808..=3417908, 3417910..=3418338, 3418340..=3418462, 3418464..=3418515, 3418517..=3418566, 3418568..=3418944, 3418946..=3419268, 3419270..=3419340, 3419342..=3419365, 3419367..=3419576, 3419578..=3419595, 3419597..=3419616, 3419618..=3419623, 3419625..=3419675, 3423615..=3423617, 3423619..=3423620, 3423622..=3423623, 3423625..=3423627, 3423629..=3423629, 3423631..=3423634, 3423636..=3423644, 3423646..=3423646, 3423648..=3423657, 3423660..=3423664, 3423666..=3423668, 3423670..=3423692, 3423695..=3423698, 3423700..=3423708, 3423712..=3423712, 3423715..=3423719, 3423721..=3423723, 3423725..=3423729, 3423731..=3423731, 3423733..=3423741, 3423743..=3423751, 3423753..=3423759, 3423761..=3423762, 3423766..=3423771, 3423773..=3423780, 3423782..=3423787, 3423789..=3423789, 3423791..=3423794, 3423796..=3423799, 3423801..=3423811, 3423813..=3423814, 3423816..=3423820, 3423822..=3423825, 3423827..=3423829, 3423831..=3423831, 3423833..=3423834, 3423836..=3423839, 3423841..=3423843, 3423845..=3423854, 3423856..=3423867, 3423869..=3423869, 3423871..=3423884, 3423886..=3423888, 3423890..=3423893, 3423896..=3423900, 3423902..=3423906, 3423908..=3423910, 3423912..=3423917, 3423919..=3423927, 3423931..=3423931, 3423933..=3423946, 3423949..=3423950, 3423952..=3423954, 3423956..=3423968, 3423970..=3423992, 3423994..=3423995, 3423998..=3424005
2024-07-01T11:54:19.971781Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:20.091893Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:20.092127Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:1006445, path:channel-6994/transfer->channel-2/transfer, toh:0-1794340, tos:2024-07-01T12:04:17.128890343Z) })
2024-07-01T11:54:20.092143Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: AF2A93CA2D779B1BA1668E429C30E340626AE0F09CBFFFB8CA3F64172DBAF2FF event="SendPacket"
2024-07-01T11:54:20.093191Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:20.093700Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-1829, client_type: ClientType(07-tendermint), consensus_height: 1-12798982 } })
2024-07-01T11:54:20.214064Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:20.214607Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:174631, path:channel-1/transfer->channel-199/transfer, toh:1-17395935, tos:NoTimestamp), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 65, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:20.400009Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: assembled batch chain=cosmoshub-4 len=1
2024-07-01T11:54:20.400055Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: broadcasting batch of 1 events chain=cosmoshub-4 count=1 height=4-21102090
2024-07-01T11:54:20.716422Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=4
2024-07-01T11:54:20.716462Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 4 events chain=osmosis-1 count=4 height=1-17395792
2024-07-01T11:54:22.374852Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:22.706440Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:54:22.875697Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:54:22.875741Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395793
2024-07-01T11:54:23.695084Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:23.937195Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:23.937853Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:397246, path:channel-326/transfer->channel-5/transfer, toh:1-9718212, tos:NoTimestamp)})
2024-07-01T11:54:23.938388Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:23.938941Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-2119, client_type: ClientType(07-tendermint), consensus_height: 1-9718068 } })
2024-07-01T11:54:24.440376Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=3
2024-07-01T11:54:24.440416Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 3 events chain=osmosis-1 count=3 height=1-17395794
2024-07-01T11:54:25.707634Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: message.module = 'ibc_channel' chain=cosmoshub-4
2024-07-01T11:54:25.707967Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:3695541, path:channel-141/transfer->channel-0/transfer, toh:no timeout, tos:2024-07-01T11:59:15.747931392Z) })
2024-07-01T11:54:25.707983Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}:ibc_channel event: tx hash: 8E336A019F65AE465E67BA1B1AFCE19135F82F1E86881999813FE277DB55B9C3 event="SendPacket"
2024-07-01T11:54:25.711726Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: message.module = 'ibc_client' chain=cosmoshub-4
2024-07-01T11:54:25.712230Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-769, client_type: ClientType(07-tendermint), consensus_height: 1-20018536 } })
2024-07-01T11:54:25.716790Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: message.module = 'ibc_channel' chain=cosmoshub-4
2024-07-01T11:54:25.717310Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:110217, path:channel-343/transfer->channel-0/transfer, toh:no timeout, tos:2078-12-30T23:57:56.700908103Z)})
2024-07-01T11:54:25.717339Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:3463359, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T12:03:50.068Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 65, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:25.719194Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: tm.event = 'NewBlock' chain=cosmoshub-4
2024-07-01T11:54:26.279973Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:26.401827Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:26.402504Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-3086, client_type: ClientType(07-tendermint), consensus_height: 1-2210434 } })
2024-07-01T11:54:26.402632Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:26.402992Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-3091, client_type: ClientType(07-tendermint), consensus_height: 1-5958504 } })
2024-07-01T11:54:26.523328Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:26.524031Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:3463360, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:59:13.1479427Z) })
2024-07-01T11:54:26.524046Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: F9DF32F67FF5E2D4E7FB0FE7D90A30C594552ACEDA2802EF1607E7FA36C67C55 event="SendPacket"
2024-07-01T11:54:26.524073Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:318670, path:channel-2/transfer->channel-19774/transfer, toh:1-17395940, tos:NoTimestamp), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 101, 121, 74, 106, 98, 50, 53, 48, 99, 109, 70, 106, 100, 70, 57, 121, 90, 88, 78, 49, 98, 72, 81, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 108, 105, 89, 49, 57, 104, 89, 50, 115, 105, 79, 105, 74, 108, 101, 85, 112, 53, 87, 108, 104, 79, 77, 87, 74, 73, 85, 87, 108, 80, 97, 85, 112, 67, 86, 86, 81, 119, 79, 85, 108, 117, 77, 68, 48, 105, 102, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:26.524137Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:26.524510Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted TimeoutPacket(TimeoutPacket { packet: seq:4773, path:channel-20082/transfer->channel-4/transfer, toh:1-5958502, tos:NoTimestamp)})
2024-07-01T11:54:26.525132Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:26.525607Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-3012, client_type: ClientType(07-tendermint), consensus_height: 0-1793341 } })
2024-07-01T11:54:26.643824Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:26.644492Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:3463361, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:58:18.8916283Z) })
2024-07-01T11:54:26.644509Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: 8CE0DEFDE942706429C15CE461610533CDAA9501586C0DDA2D215EAB0B06C3B1 event="SendPacket"
2024-07-01T11:54:26.644535Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:881267, path:channel-2/transfer->channel-6994/transfer, toh:1-17395914, tos:NoTimestamp), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 101, 121, 74, 106, 98, 50, 53, 48, 99, 109, 70, 106, 100, 70, 57, 121, 90, 88, 78, 49, 98, 72, 81, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 108, 105, 89, 49, 57, 104, 89, 50, 115, 105, 79, 105, 74, 108, 101, 85, 112, 53, 87, 108, 104, 79, 77, 87, 74, 73, 85, 87, 108, 80, 97, 85, 112, 67, 86, 86, 81, 119, 79, 85, 108, 117, 77, 68, 48, 105, 102, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:26.720101Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: assembled batch chain=cosmoshub-4 len=5
2024-07-01T11:54:26.720151Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: broadcasting batch of 5 events chain=cosmoshub-4 count=5 height=4-21102091
2024-07-01T11:54:27.146505Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=9
2024-07-01T11:54:27.146550Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 9 events chain=osmosis-1 count=9 height=1-17395795
2024-07-01T11:54:27.710245Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:54:31.720571Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: tm.event = 'NewBlock' chain=cosmoshub-4
2024-07-01T11:54:31.901267Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:32.143341Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:32.144010Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-2119, client_type: ClientType(07-tendermint), consensus_height: 1-9718069 } })
2024-07-01T11:54:32.144682Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:32.145313Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:511086, path:channel-5/transfer->channel-326/transfer, toh:1-17395939, tos:NoTimestamp), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 65, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:32.645693Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=3
2024-07-01T11:54:32.645736Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 3 events chain=osmosis-1 count=3 height=1-17395796
2024-07-01T11:54:32.713482Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:54:32.722606Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: assembled batch chain=cosmoshub-4 len=1
2024-07-01T11:54:32.722652Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: broadcasting batch of 1 events chain=cosmoshub-4 count=1 height=4-21102092
2024-07-01T11:54:33.521836Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:33.644836Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:33.645565Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-1, client_type: ClientType(07-tendermint), consensus_height: 4-21102092 } })
2024-07-01T11:54:33.765331Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:33.766062Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:3695541, path:channel-141/transfer->channel-0/transfer, toh:no timeout, tos:2024-07-01T11:59:15.747931392Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 101, 121, 74, 106, 98, 50, 53, 48, 99, 109, 70, 106, 100, 70, 57, 121, 90, 88, 78, 49, 98, 72, 81, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 108, 105, 89, 49, 57, 104, 89, 50, 115, 105, 79, 105, 74, 108, 101, 85, 112, 53, 87, 108, 104, 79, 77, 87, 74, 73, 85, 87, 108, 80, 97, 85, 112, 67, 86, 86, 81, 119, 79, 85, 108, 117, 77, 68, 48, 105, 102, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:33.766089Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:3463359, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T12:03:50.068Z)})
2024-07-01T11:54:33.766317Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:33.766562Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-3009, client_type: ClientType(07-tendermint), consensus_height: 1-19389198 } })
2024-07-01T11:54:33.884566Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:33.884951Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:3463362, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:59:26.0441795Z) })
2024-07-01T11:54:33.884962Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: 54DEE69F6F4F6DB83ADC305634E247B879938C2680A3D04FD31A57D8D890F4BC event="SendPacket"
2024-07-01T11:54:33.884982Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:43562, path:channel-3/transfer->channel-6787/transfer, toh:1-17395945, tos:NoTimestamp), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 101, 121, 74, 106, 98, 50, 53, 48, 99, 109, 70, 106, 100, 70, 57, 121, 90, 88, 78, 49, 98, 72, 81, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 108, 105, 89, 49, 57, 104, 89, 50, 115, 105, 79, 105, 74, 108, 101, 85, 112, 53, 87, 108, 104, 79, 77, 87, 74, 73, 85, 87, 108, 80, 97, 85, 112, 67, 86, 86, 81, 119, 79, 85, 108, 117, 77, 68, 48, 105, 102, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:34.386168Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=7
2024-07-01T11:54:34.386203Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 7 events chain=osmosis-1 count=7 height=1-17395797
2024-07-01T11:54:34.924276Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:35.045451Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:35.046012Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-0, client_type: ClientType(07-tendermint), consensus_height: 2-16980827 } })
2024-07-01T11:54:35.048543Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:35.049145Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:286454, path:channel-9/transfer->channel-1/transfer, toh:1-17395940, tos:NoTimestamp), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 65, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:35.550155Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=3
2024-07-01T11:54:35.550204Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 3 events chain=osmosis-1 count=3 height=1-17395798
2024-07-01T11:54:37.020740Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:37.522057Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:54:37.522113Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395799
2024-07-01T11:54:37.717023Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:54:37.937172Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: message.module = 'ibc_channel' chain=cosmoshub-4
2024-07-01T11:54:37.937864Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:3463360, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:59:13.1479427Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 65, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:37.937900Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:3463361, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:58:18.8916283Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 65, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:37.943833Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: message.module = 'ibc_client' chain=cosmoshub-4
2024-07-01T11:54:37.944493Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-259, client_type: ClientType(07-tendermint), consensus_height: 1-17395796 } })
2024-07-01T11:54:37.947579Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: message.module = 'ibc_channel' chain=cosmoshub-4
2024-07-01T11:54:37.948259Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:3695541, path:channel-141/transfer->channel-0/transfer, toh:no timeout, tos:2024-07-01T11:59:15.747931392Z)})
2024-07-01T11:54:37.948300Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:3463362, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:59:26.0441795Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 65, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:37.951240Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: message.module = 'ibc_client' chain=cosmoshub-4
2024-07-01T11:54:37.951837Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-259, client_type: ClientType(07-tendermint), consensus_height: 1-17395798 } })
2024-07-01T11:54:37.954562Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: tm.event = 'NewBlock' chain=cosmoshub-4
2024-07-01T11:54:38.787607Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:38.788290Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-1562, client_type: ClientType(07-tendermint), consensus_height: 1-14294667 } })
2024-07-01T11:54:38.908659Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:38.909341Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:695056, path:channel-75/transfer->channel-0/transfer, toh:no timeout, tos:2024-07-01T11:59:16.5936522Z)})
2024-07-01T11:54:38.955765Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: assembled batch chain=cosmoshub-4 len=7
2024-07-01T11:54:38.955801Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: broadcasting batch of 7 events chain=cosmoshub-4 count=7 height=4-21102093
2024-07-01T11:54:39.030626Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:39.032008Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:39.032554Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-3012, client_type: ClientType(07-tendermint), consensus_height: 0-1793342 } })
2024-07-01T11:54:39.152014Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:39.152693Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:3463363, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:58:34.7399004Z) })
2024-07-01T11:54:39.152714Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: 54F1BAB866492C143F5644C5C0931B022220FEA052BCE85414CEC0031ED000DC event="SendPacket"
2024-07-01T11:54:39.152743Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:881268, path:channel-2/transfer->channel-6994/transfer, toh:no timeout, tos:2024-07-01T11:58:34.739882496Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 101, 121, 74, 106, 98, 50, 53, 48, 99, 109, 70, 106, 100, 70, 57, 121, 90, 88, 78, 49, 98, 72, 81, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 108, 105, 89, 49, 57, 104, 89, 50, 115, 105, 79, 105, 74, 108, 101, 85, 112, 53, 87, 108, 104, 79, 77, 87, 74, 73, 85, 87, 108, 80, 97, 85, 112, 67, 86, 86, 81, 119, 79, 85, 108, 117, 77, 68, 48, 105, 102, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:39.152789Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:3463364, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:58:57.311676Z) })
2024-07-01T11:54:39.152795Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: 54F1BAB866492C143F5644C5C0931B022220FEA052BCE85414CEC0031ED000DC event="SendPacket"
2024-07-01T11:54:39.152814Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:881269, path:channel-2/transfer->channel-6994/transfer, toh:no timeout, tos:2024-07-01T11:58:57.311660544Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 101, 121, 74, 106, 98, 50, 53, 48, 99, 109, 70, 106, 100, 70, 57, 121, 90, 88, 78, 49, 98, 72, 81, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 108, 105, 89, 49, 57, 104, 89, 50, 115, 105, 79, 105, 74, 108, 101, 85, 112, 53, 87, 108, 104, 79, 77, 87, 74, 73, 85, 87, 108, 80, 97, 85, 112, 67, 86, 86, 81, 119, 79, 85, 108, 117, 77, 68, 48, 105, 102, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:39.152835Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:881270, path:channel-2/transfer->channel-6994/transfer, toh:1-17395936, tos:NoTimestamp), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 65, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:39.152849Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:1006444, path:channel-6994/transfer->channel-2/transfer, toh:no timeout, tos:2024-07-01T11:59:01.6933353Z)})
2024-07-01T11:54:39.654329Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=10
2024-07-01T11:54:39.654367Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 10 events chain=osmosis-1 count=10 height=1-17395800
2024-07-01T11:54:40.610769Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:41.112279Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:54:41.112323Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395801
2024-07-01T11:54:42.721940Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:54:43.413374Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:43.535268Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:43.535854Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-3012, client_type: ClientType(07-tendermint), consensus_height: 0-1793343 } })
2024-07-01T11:54:43.655875Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:43.656503Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:1006445, path:channel-6994/transfer->channel-2/transfer, toh:0-1794340, tos:2024-07-01T12:04:17.128890343Z)})
2024-07-01T11:54:43.871890Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: tm.event = 'NewBlock' chain=cosmoshub-4
2024-07-01T11:54:44.158277Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=3
2024-07-01T11:54:44.158321Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 3 events chain=osmosis-1 count=3 height=1-17395802
2024-07-01T11:54:44.873473Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: assembled batch chain=cosmoshub-4 len=1
2024-07-01T11:54:44.873515Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: broadcasting batch of 1 events chain=cosmoshub-4 count=1 height=4-21102094
2024-07-01T11:54:45.817717Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:46.060133Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:46.060905Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-1, client_type: ClientType(07-tendermint), consensus_height: 4-21102094 } })
2024-07-01T11:54:46.181104Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:46.181919Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:3463360, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:59:13.1479427Z)})
2024-07-01T11:54:46.181958Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:3463361, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:58:18.8916283Z)})
2024-07-01T11:54:46.181973Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:3463362, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:59:26.0441795Z)})
2024-07-01T11:54:46.682686Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=5
2024-07-01T11:54:46.682734Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 5 events chain=osmosis-1 count=5 height=1-17395803
2024-07-01T11:54:47.289092Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:47.289316Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:47.289479Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:272019, path:channel-19774/transfer->channel-2/transfer, toh:1-2210587, tos:NoTimestamp) })
2024-07-01T11:54:47.289491Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: 46AD07326AD57EE7CF57C89E26A364B7BC8CB87458B33BD1EEFDC65B4F6A09D7 event="SendPacket"
2024-07-01T11:54:47.726305Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:54:47.790507Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=2
2024-07-01T11:54:47.790561Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 2 events chain=osmosis-1 count=2 height=1-17395804
2024-07-01T11:54:50.270143Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:50.770888Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:54:50.770931Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395805
2024-07-01T11:54:50.805191Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: message.module = 'ibc_client' chain=cosmoshub-4
2024-07-01T11:54:50.805985Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-259, client_type: ClientType(07-tendermint), consensus_height: 1-17395801 } })
2024-07-01T11:54:50.811260Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: message.module = 'ibc_channel' chain=cosmoshub-4
2024-07-01T11:54:50.811780Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:3463363, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:58:34.7399004Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 65, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:50.811808Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:3463364, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:58:57.311676Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 65, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:50.814157Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: tm.event = 'NewBlock' chain=cosmoshub-4
2024-07-01T11:54:51.815020Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: assembled batch chain=cosmoshub-4 len=4
2024-07-01T11:54:51.815070Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: broadcasting batch of 4 events chain=cosmoshub-4 count=4 height=4-21102095
2024-07-01T11:54:52.547690Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:52.668038Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:52.668276Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:1006446, path:channel-6994/transfer->channel-2/transfer, toh:0-1793493, tos:NoTimestamp) })
2024-07-01T11:54:52.668299Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: 621949C5E5E3E3979153E8CCB47E0678B392711410C426545BD133CEE8FDF84D event="SendPacket"
2024-07-01T11:54:52.730567Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:54:53.169163Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=2
2024-07-01T11:54:53.169208Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 2 events chain=osmosis-1 count=2 height=1-17395806
2024-07-01T11:54:53.571382Z TRACE ThreadId(22) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 17395791 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 17395790 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-17395790}:query_send_packet_events{chain=osmosis-1 height=<=1-17395790 sequences=[3395732, 3395733, 3395734, 3395735, 3395736, 3395737, 3395738, 3395739, 3395740, 3395741, 3395742, 3395743, 3395744, 3395745, 3395746, 3395747, 3395748, 3395749, 3395750, 3395751, 3395752, 3395753, 3395754, 3395755, 3395756, 3395757, 3395758, 3395759, 3395760, 3395761, 3395762, 3395763, 3395764, 3395765, 3395766, 3395767, 3395768, 3395769, 3395770, 3395771, 3395772, 3395773, 3395774, 3395775, 3395776, 3395777, 3395778, 3395779, 3395780, 3395781]}: start_block_events []
2024-07-01T11:54:53.571424Z TRACE ThreadId(22) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 17395791 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 17395790 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-17395790}:query_send_packet_events{chain=osmosis-1 height=<=1-17395790 sequences=[3395732, 3395733, 3395734, 3395735, 3395736, 3395737, 3395738, 3395739, 3395740, 3395741, 3395742, 3395743, 3395744, 3395745, 3395746, 3395747, 3395748, 3395749, 3395750, 3395751, 3395752, 3395753, 3395754, 3395755, 3395756, 3395757, 3395758, 3395759, 3395760, 3395761, 3395762, 3395763, 3395764, 3395765, 3395766, 3395767, 3395768, 3395769, 3395770, 3395771, 3395772, 3395773, 3395774, 3395775, 3395776, 3395777, 3395778, 3395779, 3395780, 3395781]}: tx_events []
2024-07-01T11:54:53.571432Z TRACE ThreadId(22) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 17395791 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 17395790 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-17395790}:query_send_packet_events{chain=osmosis-1 height=<=1-17395790 sequences=[3395732, 3395733, 3395734, 3395735, 3395736, 3395737, 3395738, 3395739, 3395740, 3395741, 3395742, 3395743, 3395744, 3395745, 3395746, 3395747, 3395748, 3395749, 3395750, 3395751, 3395752, 3395753, 3395754, 3395755, 3395756, 3395757, 3395758, 3395759, 3395760, 3395761, 3395762, 3395763, 3395764, 3395765, 3395766, 3395767, 3395768, 3395769, 3395770, 3395771, 3395772, 3395773, 3395774, 3395775, 3395776, 3395777, 3395778, 3395779, 3395780, 3395781]}: end_block_events []
2024-07-01T11:54:53.571511Z INFO ThreadId(26) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 17395791 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 17395790 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-17395790}: pulled packet data for 0 events out of 50 sequences: 3395732..=3395781; events.total=23757 events.left=23707
2024-07-01T11:54:54.244497Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:54.746271Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:54:54.746306Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395807
2024-07-01T11:54:56.793982Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: tm.event = 'NewBlock' chain=cosmoshub-4
2024-07-01T11:54:57.393693Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:57.733578Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:54:57.794935Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: assembled batch chain=cosmoshub-4 len=1
2024-07-01T11:54:57.794975Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: broadcasting batch of 1 events chain=cosmoshub-4 count=1 height=4-21102096
2024-07-01T11:54:57.894908Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:54:57.894980Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395808
2024-07-01T11:54:59.155443Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:59.398303Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:59.399067Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-1, client_type: ClientType(07-tendermint), consensus_height: 4-21102096 } })
2024-07-01T11:54:59.519024Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:59.519828Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:3463363, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:58:34.7399004Z)})
2024-07-01T11:54:59.519862Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:3463364, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:58:57.311676Z)})
2024-07-01T11:55:00.020810Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=4
2024-07-01T11:55:00.020851Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 4 events chain=osmosis-1 count=4 height=1-17395809
2024-07-01T11:55:01.936597Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:55:02.178261Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:55:02.178936Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:272019, path:channel-19774/transfer->channel-2/transfer, toh:1-2210587, tos:NoTimestamp)})
2024-07-01T11:55:02.179565Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:55:02.180152Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-3086, client_type: ClientType(07-tendermint), consensus_height: 1-2210440 } })
2024-07-01T11:55:02.681642Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=3
2024-07-01T11:55:02.681679Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 3 events chain=osmosis-1 count=3 height=1-17395810
2024-07-01T11:55:02.737250Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:55:02.881140Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: tm.event = 'NewBlock' chain=cosmoshub-4
2024-07-01T11:55:03.325513Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:55:03.826986Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:55:03.827028Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395811
2024-07-01T11:55:03.882122Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: assembled batch chain=cosmoshub-4 len=1
2024-07-01T11:55:03.882165Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: broadcasting batch of 1 events chain=cosmoshub-4 count=1 height=4-21102097
2024-07-01T11:55:05.150059Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:55:05.651584Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:55:05.651654Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395812
2024-07-01T11:55:07.742155Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:55:08.249223Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:55:08.249821Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:55:08.250176Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-2704, client_type: ClientType(07-tendermint), consensus_height: 1-7409828 } })
2024-07-01T11:55:08.371065Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:55:08.371562Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:69947, path:channel-38946/transfer->channel-1/transfer, toh:no timeout, tos:2024-07-01T12:00:00.208658Z) })
2024-07-01T11:55:08.371580Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: 7930F302C61795FB709D7F2D92BC20FB6A99DC4F5C4DC40BA7567FA36B7064E9 event="SendPacket"
2024-07-01T11:55:08.371606Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:220305, path:channel-1/transfer->channel-750/transfer, toh:1-17395959, tos:NoTimestamp), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 101, 121, 74, 106, 98, 50, 53, 48, 99, 109, 70, 106, 100, 70, 57, 121, 90, 88, 78, 49, 98, 72, 81, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 108, 105, 89, 49, 57, 104, 89, 50, 115, 105, 79, 105, 74, 108, 101, 85, 112, 53, 87, 108, 104, 79, 77, 87, 74, 73, 85, 87, 108, 80, 97, 85, 112, 67, 86, 86, 81, 119, 79, 85, 108, 117, 77, 68, 48, 105, 102, 81, 61, 61, 34, 125 ] })
2024-07-01T11:55:08.492184Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:55:08.492856Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-3086, client_type: ClientType(07-tendermint), consensus_height: 1-2210441 } })
2024-07-01T11:55:08.494246Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:55:08.494882Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:1006447, path:channel-6994/transfer->channel-2/transfer, toh:no timeout, tos:2024-07-01T11:59:53.5405522Z) })
2024-07-01T11:55:08.494902Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: 093B3CE7D6F773A00B5FFB48E0804BFD900FB165E8928E41AED88DC1DE5F0C95 event="SendPacket"
2024-07-01T11:55:08.494929Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:318671, path:channel-2/transfer->channel-19774/transfer, toh:1-17395957, tos:NoTimestamp), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 101, 121, 74, 106, 98, 50, 53, 48, 99, 109, 70, 106, 100, 70, 57, 121, 90, 88, 78, 49, 98, 72, 81, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 108, 105, 89, 49, 57, 104, 89, 50, 115, 105, 79, 105, 74, 108, 101, 85, 112, 53, 87, 108, 104, 79, 77, 87, 74, 73, 85, 87, 108, 80, 97, 85, 112, 67, 86, 86, 81, 119, 79, 85, 108, 117, 77, 68, 48, 105, 102, 81, 61, 61, 34, 125 ] })
2024-07-01T11:55:08.613252Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:55:08.613846Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-3012, client_type: ClientType(07-tendermint), consensus_height: 0-1793345 } })
2024-07-01T11:55:08.615150Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:55:08.615724Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:3463365, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:59:29.158915Z) })
2024-07-01T11:55:08.615740Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: A1263FA3B6B3F69BBB13C4DA90122CBEB96D89F06D414F7F7B2A521F58A1F265 event="SendPacket"
2024-07-01T11:55:08.615766Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:881271, path:channel-2/transfer->channel-6994/transfer, toh:no timeout, tos:2024-07-01T11:59:29.158901248Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 101, 121, 74, 106, 98, 50, 53, 48, 99, 109, 70, 106, 100, 70, 57, 121, 90, 88, 78, 49, 98, 72, 81, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 108, 105, 89, 49, 57, 104, 89, 50, 115, 105, 79, 105, 74, 108, 101, 85, 112, 53, 87, 108, 104, 79, 77, 87, 74, 73, 85, 87, 108, 80, 97, 85, 112, 67, 86, 86, 81, 119, 79, 85, 108, 117, 77, 68, 48, 105, 102, 81, 61, 61, 34, 125 ] })
2024-07-01T11:55:08.615798Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:1006448, path:channel-6994/transfer->channel-2/transfer, toh:no timeout, tos:2024-07-01T12:11:44.661126659Z) })
2024-07-01T11:55:08.615803Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: A1263FA3B6B3F69BBB13C4DA90122CBEB96D89F06D414F7F7B2A521F58A1F265 event="SendPacket"
2024-07-01T11:55:08.615817Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:881272, path:channel-2/transfer->channel-6994/transfer, toh:no timeout, tos:2024-07-01T12:06:37.589Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 101, 121, 74, 106, 98, 50, 53, 48, 99, 109, 70, 106, 100, 70, 57, 121, 90, 88, 78, 49, 98, 72, 81, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 108, 105, 89, 49, 57, 104, 89, 50, 115, 105, 79, 105, 74, 108, 101, 85, 112, 53, 87, 108, 104, 79, 77, 87, 74, 73, 85, 87, 108, 80, 97, 85, 112, 67, 86, 86, 81, 119, 79, 85, 108, 117, 77, 68, 48, 105, 102, 81, 61, 61, 34, 125 ] })
2024-07-01T11:55:09.116935Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=12
2024-07-01T11:55:09.116978Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 12 events chain=osmosis-1 count=12 height=1-17395813
2024-07-01T11:55:09.133164Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: tm.event = 'NewBlock' chain=cosmoshub-4
2024-07-01T11:55:09.281800Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:55:09.783435Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:55:09.783483Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395814
2024-07-01T11:55:10.134346Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: assembled batch chain=cosmoshub-4 len=1
2024-07-01T11:55:10.134387Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: broadcasting batch of 1 events chain=cosmoshub-4 count=1 height=4-21102098
2024-07-01T11:55:11.786030Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:55:12.286728Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:55:12.286772Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395815
2024-07-01T11:55:12.746255Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:55:13.097484Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:55:13.599145Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:55:13.599183Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395816
Ok so it seems the clear_on_start does indeed block the relaying. This is because this part should clear any pending packets having the normal relaying run concurrently will likely generate redundant packet relaying.
I was under the impression that the clear_interval shouldn't block the relaying, but I will look into this as well.
@ljoss17 can packets clearing interfere with Hermes functioning properly? (my thought was they might submit transaction and mess with account sequence, but not sure if if it's the case or if there's more caveats)
Clear on start will first finish trying to clear all packets, due to the high number of packets in your case the instance is "stuck" at the clearing phase and that is why it isn't relaying. Could you try adding the pending packets to the excluded_sequences and see if the relaying works even with the clearing enabled?
For interval clearing, if it happens concurrently to the relaying, one of the two tentatives to relay might result in redundant packet error (either the relaying is faster or the clearing) if there is a new packet when clearing is triggered.
@ljoss17 one more thing I just discovered: so I used the same config as above, with trace logging, but without clear_on_start (but with clear_interval), and it seems to also not relay anything since the interval clearing was executed.
It did some transactions when starting up, but then it stopped, I usually see way more broadcast errors when the app is working correctly, here's what I see now:
Here's the same metric for the last 6 hours:
Basically it has 3 periods:
- till ~14:55 (JFYI this is from today and I'm in UTC+3) - the relayer is working properly because I disabled the packets clearing both on startup and interval
- till ~16:30 - there's no data apparently because there's no such metric, this is when I enabled both startup and interval clearing, and it didn't finish the startup, so it never broadcasted a transaction that error (actually it didn't broadcast any transaction at all, either successful or failed)
- afterwards - startup clearing is disabled, interval clearing is enabled; apparently once the interval clearing is in action, it also doesn't relay.
Here are the trace logs for the period since 16:30, hope it would be of any help: https://gist.github.com/freak12techno/d6688f271f34ec6dc70053d938375062
For excluded_sequences, gonna check it out a bit later.
@ljoss17 just to clarify: my biggest concern here is not the packets being not relayed correctly (this is likely either the node being misconfigured, or blocks being pruned), but rather Hermes not doing anything once the packets are being cleared. I think that might be disturbing if let's say there are indeed some packets being pending, a lot of them, so clearing would take some time, and during that time Hermes won't function properly. This specific case is likely something I can resolve; problem is once this happens again, it'll (apparently) block Hermes from doing anything else.
I see, for the clear on start I will look into it to see if there is a way to cleanly split the clearing part and relaying part so that both can run concurrently without interference. For the clear interval it might be a bit more tricky but it should also be possible.
If the clearing and standard relaying have a clear separation it will be easier to have them run concurrently.
@ljoss17 okay so, the excluded_sequences config part did the trick apparently, therefore I confirm it's working with it.