monorepo icon indicating copy to clipboard operation
monorepo copied to clipboard

Published 20 hours ago • verified publisher icon connext

Monitor and Alert when routers not correctly filling transfers

Open preethamr opened this issue 2 years ago • 74 comments

Problem

When a properly setup active router stops bidding, we don't know of it, until operator (we or router) checks.

Impact

Router operator miss on potential revenue, possibly transfers not getting boosted.

Proposed Solution

Monitor every active router, and alert when it stops sending successful bids, for more than a threshold amount of time

Acceptance Criteria

From an end user perspective, the below criteria must be met to consider this done:

  • [ ] Every active router is monitored
  • [ ] Alert sent to a Discord alert channel

preethamr avatar Oct 07 '23 05:10 preethamr

Examples of transactions impacted by routers not bidding:

  • https://app.zenhub.com/workspaces/core-engineering-6272a34e0368a200141cd949/issues/gh/connext/monorepo/5510
  • https://app.zenhub.com/workspaces/core-engineering-6272a34e0368a200141cd949/issues/gh/connext/monorepo/5543
  • https://app.zenhub.com/workspaces/core-engineering-6272a34e0368a200141cd949/issues/gh/connext/monorepo/5482
  • https://app.zenhub.com/workspaces/core-engineering-6272a34e0368a200141cd949/issues/gh/connext/monorepo/5478

just-a-node avatar Jan 13 '24 00:01 just-a-node

Transaction: https://connextscan.io/tx/0xae6fd740c77e5d65888df0a1d1500ba751a11ca0d0a4a05c36651688783bd553?source=search Router: https://connextscan.io/router/0x58507fed0cb11723dfb6848c92c59cf0bbeb9927 Ticket: https://discord.com/channels/454734546869551114/1196852916334641302/1196853006247928041

Tommy-R8 avatar Jan 16 '24 16:01 Tommy-R8

Transaction:

https://connextscan.io/tx/0x544f0fff95308c4502ee9273ac2a4c3e91b1e0862b2495a7392367f3353b1a3b

Router

https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4

Ticket:

https://discord.com/channels/454734546869551114/1196981228457426944

MinistroDolar avatar Jan 17 '24 01:01 MinistroDolar

Transactions from Optimism to Polygon:

https://connextscan.io/tx/0xc8e4c1cdcd852deb3eef1d140ce8e45eb395ddf348258b897fc690b6f2a88a50 https://connextscan.io/tx/0x64bbd9125bfbb9e46aea9bbd3047348e3fd76373587a0b03463b4320134aa826

Routers on Polygon:

https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4 https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba

Transactions from Optimism to Arbitrum:

https://connextscan.io/tx/0xfe5d464d5b6d9c2f5c8b0108a3a9fdacf7259c74114c0a043ead731d2b90709f

Routers on Arbitrum:

https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4 https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba

Offline router on arbitrum:

https://connextscan.io/router/0x49a9e7ec76bc8fdf658d09557305170d9f01d2fa

Ticket on Discord: https://discord.com/channels/454734546869551114/1197943725003915364/1197943801570926664

Tommy-R8 avatar Jan 19 '24 16:01 Tommy-R8

Transactions from Optimism to Arbitrum:

https://connextscan.io/tx/0x31a819db847bbe0bf1c3bc32d39961c8084937043ada2ec2c8dd26f1dcfb60be https://connextscan.io/tx/0xa74cc5d97c2e181530d282c15e50bd395a2c85926c81e008f73e44114ca9e3c6

Routers on Arbitrum:

https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4 https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba

Offline router on arbitrum:

https://connextscan.io/router/0x49a9e7ec76bc8fdf658d09557305170d9f01d2fa

Ticket on Discord: https://discord.com/channels/454734546869551114/1197944382502993970/1197944398047100938

Tommy-R8 avatar Jan 19 '24 16:01 Tommy-R8

Large txn from BNB Chain to Gnosis:

https://connextscan.io/tx/0x165434fae546628bf17b84dce493f79aa5160a0713635de89f9971af8c3fd325

Gnosis router not bidding:

https://connextscan.io/router/0x048a5ecc705c280b2248aeff88fd581abbeb8587

MinistroDolar avatar Jan 22 '24 17:01 MinistroDolar

Transaction: https://connextscan.io/tx/0xde18d913300013af0e92c764ca3ae8ea82c473d427873fad9883f7213f255a66

Router: https://connextscan.io/router/0x048a5ecc705c280b2248aeff88fd581abbeb8587

Ticket: https://discord.com/channels/454734546869551114/1199301697118416993/1199301968397602886

Tommy-R8 avatar Jan 23 '24 13:01 Tommy-R8

Transaction: https://connextscan.io/tx/0x15100b04549b63e93be0fe62ef01ea771d4505ac899132b1bbb27dd306935c14

Router: https://connextscan.io/router/0x048a5ecc705c280b2248aeff88fd581abbeb8587

Ticket: https://discord.com/channels/454734546869551114/1199292091004624946/1199292226568724520

Tommy-R8 avatar Jan 23 '24 13:01 Tommy-R8

Transaction: https://connextscan.io/tx/0xfe54645511f482ba98532c5437ab52c773e4a9a1840224f95f30d31e7adc5f65

Router: https://connextscan.io/router/0x048a5ecc705c280b2248aeff88fd581abbeb8587

Ticket: https://discord.com/channels/454734546869551114/1199445475749789756/1199445960472924191

Tommy-R8 avatar Jan 23 '24 20:01 Tommy-R8

Transaction: https://connextscan.io/tx/0x29e1d009280003c59552a700e7c60419dc277b97d55798bc3f7b761bc8ea6ecf

Router: https://connextscan.io/router/0x58507fed0cb11723dfb6848c92c59cf0bbeb9927

Ticket: https://discord.com/channels/454734546869551114/1200519660160942211/1200519707770503300

Tommy-R8 avatar Jan 26 '24 19:01 Tommy-R8

Transaction: https://connextscan.io/tx/0xe4f6447eda37bef24a8798f3a7b2c415ecd35e35909d573192d4da9bee9a562c

Router: https://connextscan.io/router/0x8cb19ce8eedf740389d428879a876a3b030b9170

Ticket: https://discord.com/channels/454734546869551114/1200609226733670462/1200614121666773002

Tommy-R8 avatar Jan 27 '24 01:01 Tommy-R8

Route: USDC BNB to Linea Transaction: https://connextscan.io/tx/0xb647f4fb648962dfdf9a336abe3c93058d8c0a8c94cff7fbc39757917a0f301d Router: https://connextscan.io/router/0x58507fed0cb11723dfb6848c92c59cf0bbeb9927 Ticket: https://discord.com/channels/454734546869551114/1200718193581371442

ValerywArz avatar Jan 27 '24 08:01 ValerywArz

Transaction: https://connextscan.io/tx/0x1b1c292906cf0d951e1f259c6ae20cb6c56e14119d5c8b89a4486735cf013bc9

Router: https://connextscan.io/router/0x8cb19ce8eedf740389d428879a876a3b030b9170

Ticket: https://discord.com/channels/454734546869551114/1200963969960132719/1200964049257644043

Tommy-R8 avatar Jan 28 '24 00:01 Tommy-R8

https://connextscan.io/tx/0xc572d6076b0ca8cf95a1f1f3819fb35e6e6a07662215ee665839cffa1c673319

from Karpatkey didnt go through

alexwhte avatar Jan 31 '24 19:01 alexwhte

Transaction: https://connextscan.io/tx/0xda93e386b259a4ef6a548815b2815135b96d67865a4e0a842201461ee86299f1

Routers: https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba https://connextscan.io/router/0x33b2ad85f7dba818e719fb52095dc768e0ed93ec https://connextscan.io/router/0x7ce49752ffa7055622f444df3c69598748cb2e5f

Ticket: 120 on Intercom

Tommy-R8 avatar Feb 01 '24 17:02 Tommy-R8

Transaction: https://connextscan.io/tx/0xec1d64082f907a140e5ed12de01ef55b264bdb44bb65de0838568cb8260e8200

Router: https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4

Ticket: https://discord.com/channels/454734546869551114/1202926909541978112/1202927071991697438

Tommy-R8 avatar Feb 02 '24 12:02 Tommy-R8

RCA: https://www.notion.so/Gnosis-Asman-Router-Issues-b9a864625a394f06870d206fbf8aa80e Initial root cause for 3:

  • Routers poll loop around retryXcalls likely freeze on connection mq.publish https://github.com/connext/monorepo/blob/4fb5d1ea4597ff61604b73fe6dc32a9912c5a97a/packages/agents/router/src/tasks/publisher/operations/retryXCalls.ts#L95-L115

Overall Cause:

Our routers don’t run rabbitMQ in a HA environment.

preethamr avatar Feb 03 '24 07:02 preethamr

Transaction: https://connextscan.io/tx/0xfd1b7fc34491d0ada8afde841d3491e261e2e96fa519ba71557d125c2fa25c5f

Routers: https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba https://connextscan.io/router/0x6fd84ba95525c4ccd218f2f16f646a08b4b0a598

Ticket: https://discord.com/channels/454734546869551114/1204446970480042035

MinistroDolar avatar Feb 06 '24 16:02 MinistroDolar

Transaction:

https://connextscan.io/tx/0xfd1b7fc34491d0ada8afde841d3491e261e2e96fa519ba71557d125c2fa25c5f

Routers:

https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba https://connextscan.io/router/0x6fd84ba95525c4ccd218f2f16f646a08b4b0a598

Ticket:

https://discord.com/channels/454734546869551114/1204446970480042035

Adding 2 more. Same destination chain and routers

Transaction: https://connextscan.io/tx/0xbf10c1f5ff68959ed45ea595d28b719a5c6d80a67cb9409005117d66172e1928

Ticket: https://discord.com/channels/454734546869551114/1205144727813619744

Transaction: https://connextscan.io/tx/0x50f3bd5d77009b420e095e3647f1839e2ffd4d3ec0b71bc3d3dcf431109b0a14

Ticket: https://discord.com/channels/454734546869551114/1205235577650413639

MinistroDolar avatar Feb 08 '24 13:02 MinistroDolar

Case 1: 3.2 Metis from Andromeda to Mainnet

Transaction: https://connextscan.io/tx/0x48781fdf83b7036b46a7993b685d1c37a2341f5ee36f65410035293e7f6a4c73

Routers: https://connextscan.io/router/0x8cb19ce8eedf740389d428879a876a3b030b9170

Ticket: https://discord.com/channels/454734546869551114/1207316640471977994

Case 2: 304 USDT from BNB to Arbitrum

Transaction: https://connextscan.io/tx/0x287a9edb884664bcc5d5d71843593f43fa852d00a04178fe2446f0a9949b3d80

Routers that could fulfill the txn: https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4 https://connextscan.io/router/0x49a9e7ec76bc8fdf658d09557305170d9f01d2fa https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba https://connextscan.io/router/0xf26c772c0ff3a6036bddabdaba22cf65eca9f97c

Ticket: https://discord.com/channels/454734546869551114/1207336207097528360

MinistroDolar avatar Feb 14 '24 15:02 MinistroDolar

Transaction: https://connextscan.io/tx/0xdcae3368f11ee562ba51dbc57ef0b62d4eaaa3d1a8ab6d675e9bef3606863905 Ticket 152 on Intercom

Transaction: https://connextscan.io/tx/0x032a1173d7e868b3cd0376d537253bdb3401984eba0efc85e64bd6b0c8715106 Ticket 154 on Intercom

Transaction: https://connextscan.io/tx/0xad762da074eeafd097160ea9af9fd926e0571f4957d6478c46b69433ef17864 Ticket 150 on Intercom

Transaction: https://connextscan.io/tx/0x0440af350352d17d8b9468b538f62e646857139b724ed16005e492f5f8a02942 Ticket 153 on Intercom

Tommy-R8 avatar Feb 14 '24 18:02 Tommy-R8

Transaction: https://connextscan.io/tx/0xdcae3368f11ee562ba51dbc57ef0b62d4eaaa3d1a8ab6d675e9bef3606863905 Ticket 152 on Intercom

Transaction: https://connextscan.io/tx/0x032a1173d7e868b3cd0376d537253bdb3401984eba0efc85e64bd6b0c8715106 Ticket 154 on Intercom

Transaction: https://connextscan.io/tx/0xad762da074eeafd097160ea9af9fd926e0571f4957d6478c46b69433ef17864 Ticket 150 on Intercom

Transaction: https://connextscan.io/tx/0x0440af350352d17d8b9468b538f62e646857139b724ed16005e492f5f8a02942 Ticket 153 on Intercom

All of these transfers received router bids, but after the gelato relayer address migration, they are awaiting retry from routers. FYI routers progressively decrease their retry frequency.

preethamr avatar Feb 14 '24 18:02 preethamr

Transaction: (WETH from mainnet to ARB) https://connextscan.io/tx/0x0e001487c0f1adedc6d5fd99702a6e1a86bd80e7b6bd659f2c8eaeace5514ea1

Router: https://connextscan.io/router/0x49a9e7ec76bc8fdf658d09557305170d9f01d2fa

Ticket: https://discord.com/channels/454734546869551114/1209217534373531718

Idk why but seems that there is only 1 router for WETH now on Arbitrum

MinistroDolar avatar Feb 19 '24 19:02 MinistroDolar

Transaction: (WETH from mainnet to ARB) https://connextscan.io/tx/0x0e001487c0f1adedc6d5fd99702a6e1a86bd80e7b6bd659f2c8eaeace5514ea1

Router: https://connextscan.io/router/0x49a9e7ec76bc8fdf658d09557305170d9f01d2fa

Ticket: https://discord.com/channels/454734546869551114/1209217534373531718

Idk why but seems that there is only 1 router for WETH now on Arbitrum

Adding 1 more txn from Gnosis to Arbitrum (it's sad that a txn lower than 0.006 eth can't achieve fast path) https://connextscan.io/tx/0xe6f14c5f9e0cb4eb71ae611ac3baf1b6214284d7d6b6f9b88f3a19ab30acdf47

MinistroDolar avatar Feb 19 '24 20:02 MinistroDolar

Transaction: (USDC from Linea to BNB) https://connextscan.io/tx/0xa2e053242596cdc3cfd987d03315feed9676b4d59ea2648374ceaea7b3290398

Routers: (this txn should be completed by more than 1 router bc of it size) https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4 https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba https://connextscan.io/router/0x33b2ad85f7dba818e719fb52095dc768e0ed93ec https://connextscan.io/router/0x49a9e7ec76bc8fdf658d09557305170d9f01d2fa https://connextscan.io/router/0x58507fed0cb11723dfb6848c92c59cf0bbeb9927

Ticket: https://discord.com/channels/454734546869551114/1209255352139251792

MinistroDolar avatar Feb 19 '24 22:02 MinistroDolar

Transaction (ETH from BNB to ETH): https://connextscan.io/tx/0xded162388328a7dfc5177a2346bb8843a99aba1a8e7b97299bf517e07fd1c4be

Routers: https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4

Ticket 173 on Intercom

Tommy-R8 avatar Feb 19 '24 22:02 Tommy-R8

Transaction (USDT from BNB to Metis): https://connextscan.io/tx/0x66a9fe845c3bbc07fdce121ddf9a7496950e47cf730318649f03515bb9b1dc06

Router: https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4

Ticket 185 on Intercom

Tommy-R8 avatar Feb 21 '24 22:02 Tommy-R8

Transaction: https://connextscan.io/tx/0x2508942ff99ea2949322cd8024db25e9d5b328124661bc8fc5949932aa77ed03 Router: https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4 Ticket: https://discord.com/channels/454734546869551114/1212515483769905194/1212516276866650112

Tommy-R8 avatar Feb 29 '24 00:02 Tommy-R8

Transaction (USDT from Optimism to Metis): https://connextscan.io/tx/0xff8052f6ab4259542ddf3ba9c7b352d13a9ff14014b5c22fe50d16d93177bffc

Router: https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4

Ticket: https://discord.com/channels/454734546869551114/1212731078805291109

MinistroDolar avatar Feb 29 '24 12:02 MinistroDolar

Transaction (USDT from Base to Arb): https://connextscan.io/tx/0x5f8db76a9ce28053c74728146122a0eb7cbeb1701797c8725fd3727ddc4b804e

Router: https://connextscan.io/router/0xeca085906cb531bdf1f87efa85c5be46aa5c9d2c https://connextscan.io/router/0x49a9e7ec76bc8fdf658d09557305170d9f01d2fa https://connextscan.io/router/0xbe7bc00382a50a711d037eaecad799bb8805dfa8

Ticket: https://discord.com/channels/454734546869551114/1215046089229737994

MinistroDolar avatar Mar 06 '24 21:03 MinistroDolar