hyperlane-monorepo icon indicating copy to clipboard operation
hyperlane-monorepo copied to clipboard

Epic: Agent Logging Improvements

Open tkporter opened this issue 2 years ago • 3 comments

I think it's been a while since we spent some time poking through agent logs and making sure they're easy to understand, not overly polluted, and correctly formatted. In debugging I've noticed a few times that it's been really hard to navigate

Some specific things:

  • These AWS / ethers related debug logs are too chatty and unnecessary imo. I don't know if we can easily change these to trace or just omit them entirely, but it'd be nice to get rid of them...
    • https://cloudlogging.app.goo.gl/YhedR3DS8L7WmpWM6
  • There are some logs in here that aren't friendly for structured logging that we should change. i.e. no string formatting
    • https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/rust/chains/hyperlane-sealevel/src/mailbox.rs#L492-L513
  • Just spend some time scrolling through logs for omniscient relayer, validators, and the scraper in GCP / kubernetes and see if anything stands out.
  • I personally find the matching lists that are logged in spans super frequently for the rc context to be really hard to navigate and polluting
  • We can probably remove the return value instrumentation here https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/2d8fced77a7b8e638163684a67355cba7cbe7a66/rust/chains/hyperlane-sealevel/src/mailbox.rs#L284 (called out by Nam and Kurtosis https://discord.com/channels/935678348330434570/935678739663192184/1153456314819825735)
  • If no new logs are present, https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/71e8988ccd095d7d0a89ebcacf2324918be2e9da/rust/hyperlane-base/src/contract_sync/mod.rs#L69 can be less frequent or possibly debug (called out by Nam and Kurtosis https://discord.com/channels/935678348330434570/935678739663192184/1153456494558314517)
  • From Yorke: I have no idea how this works but it spits out really ugly logs https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/6eca260452dd77b2ae8e8564041725815a7fabc6/rust/agents/relayer/src/msg/pending_operation.rs#L119
### Tasks
- [ ] https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3496
- [ ] https://github.com/hyperlane-xyz/issues/issues/648
- [ ] https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3146
- [ ] https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2233
- [ ] https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3375
- [ ] Add clear log about agents starting up so we can more easily search for restarts
- [ ] https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3823

tkporter avatar Aug 17 '23 13:08 tkporter

trevor has started some of the work here

avious00 avatar Sep 13 '23 15:09 avious00

I deleted the Solana files (part of this issue before, but now removed), but haven't really addressed any of the logging stuff. I think may make sense to give to Dan / Mattie as availability opens up

tkporter avatar Sep 19 '23 09:09 tkporter