lndhub.go icon indicating copy to clipboard operation
lndhub.go copied to clipboard

Improve sentry integration

Open bumi opened this issue 2 years ago • 9 comments

Currently sentry reports the line where we call CaptureException and not where the error actually happened. I think it should be possible to improve this that Sentry tells us the line where the error happend.

also to monitor the application performance we should add opentelementry instrumentation for echo.

https://docs.sentry.io/platforms/go/guides/echo/performance/instrumentation/opentelemetry/ https://uptrace.dev/opentelemetry/instrumentations/go-echo.html#echo-instrumentation https://uptrace.dev/opentelemetry/go-tracing.html

bumi avatar Jan 05 '23 18:01 bumi

Just a wild guess:

https://github.com/getAlby/lndhub.go/blob/main/main.go#L113

According to the go echo guide you should first init sentry, then add all the middlewares, and then the sentryecho middleware.

image

https://docs.sentry.io/platforms/go/guides/echo/

@kiwiidb Do you think that could potentially be the issue here? Also I don't really get the defer part of our integration. 🤔

reneaaron avatar Jan 05 '23 20:01 reneaaron

Currently there is no performance integration that can easily be added and would allow you to see query times, response times, etc. There are 2 methods to do that right now:

  • Custom instrumentation
  • OpenTelemetry (Currently beta, might be worth checking out soon)

See the docs for more info https://docs.sentry.io/platforms/go/guides/echo/performance/instrumentation/

reneaaron avatar Feb 15 '23 23:02 reneaaron

do we still need this? opentelemetry is now in with datadog, isn't it?

bumi avatar Mar 02 '23 12:03 bumi

I just set up Sentry and added the SENTRY_DSN config, but it doesn't seem to be sending events. Maybe this particular type of error isn't being sent intentionally?

{"level":"error","time":"2023-03-25T00:03:26Z","message":"Error tracking payment with hash 123add327f37c378a2006e6123c6ac6daa6b922fe2f53aa18b2fe2eb1d8257b8: rpc error: code = NotFound desc = payment isn't initiated"}

Since someone above mentioned potential issues with how Sentry is integrated into the code, I thought I'd put this question here instead of opening a new issue.

raucao avatar Mar 25 '23 00:03 raucao

Logs are not sent to Sentry, only errors / unhandled exceptions currently. Would you want to see this particular error in Sentry, too? Currently only some of the errors seem to be captured:

https://github.com/getAlby/lndhub.go/blob/main/lib/service/checkpayments.go#L77

The issue itself is about integrating more detailed logs about e.g. database queries and such, the general integration should work fine.

reneaaron avatar Mar 25 '23 20:03 reneaaron

OK, thanks.

Would you want to see this particular error in Sentry, too?

Yes, I think most, if not all, errors should be sent, because as an operator, you can ignore specific ones on the Sentry side and only keep the ones you care about.

I haven't looked into the new performance tracking features of Sentry yet, but those could also be interesting for improving database queries, indexes and such.

raucao avatar Mar 26 '23 07:03 raucao

I found another thing that would be useful to improve: Currently, there is no environment set for Sentry.

I would propose to set production as environment by default, and perhaps add another config key SENTRY_ENV or similar for people who may want to run multiple instances in e. staging and prod.

raucao avatar Apr 03 '23 13:04 raucao

That would certainly make sense. Any chance you want to prepare a PR?

reneaaron avatar Apr 03 '23 14:04 reneaaron

Just contributing suggestions and feedback for now, since I currently have no cycles free for contributing code here.

raucao avatar Apr 03 '23 15:04 raucao