epic-stack
epic-stack copied to clipboard
epicweb-dev
Implement Logging solution
In keeping with our avoid third party services principle, I'd like to find whether there's a good way to keep logs for easy retrieval and analysis later.
I've thought about doing this as a process running on the same VM, but that will miss startup logging and runs into limitations with combining logs from multiple instances. So we'll probably want this as a separate service.
However, I'd like to avoid signing up for an account on another service and I'd instead like to run something ourselves on Fly which I expect would be cheaper than using a hosted service and shouldn't take any more time. Fly has https://github.com/superfly/fly-log-shipper which allows for easy forwarding of logs to a running service. I think we just need to look for a good tool that allows for reviewing logs.
I would be willing to require signup for another service if it is sufficiently complex or costly to deploy and maintain our own running service.
Suggestions are welcome.
More info on logging with fly: https://fly.io/docs/going-to-production/monitoring/exporting-logs/#the-fly-log-shipper
Looks like the recommended solution is to use log shipper as a separate app. So now we need to decide a provider. Hopefully one we can reasonably run ourselves.
Thinking about this more, I think I'd like it if our error tracking and logging service were the same. I don't know if sentry is capable of this. New Relic seems to have discontinued its error tracking service. However I think Log Rocket has both. More investigation is needed.
Looks like Log Rocket only handles the frontend aspect. Bummer. I'm thinking we may not be able to avoid using separate services for logging and error reporting.
Thinking about this more, I think I'd like it if our error tracking and logging service were the same. I don't know if sentry is capable of this. New Relic seems to have discontinued its error tracking service. However I think Log Rocket has both. More investigation is needed.
I thought that we were talking about 2 separate services from the start. Sentry is a must-have for error tracking so I think that we need something more for log management.
We can do a sheet with all the free possibilities to analyze the pros and cons. If I'm not mistaken, we are currently in search for:
- Run something ourselves on Fly.
- It should be cheaper than using a hosted service and shouldn't require any additional time
- Allows for reviewing logs
- The easier it is to implement and maintain, the better, whether it is self-hosted or provided by a third-party service.
- No credit card is required
Isn't it better to choose something from Vector's Sink list?
I think I'd prefer to look into the supported providers here: https://github.com/superfly/fly-log-shipper
We use fly-log-shipper -> Axiom - their free tier is pretty generous at 500GB for 30 days and the interface has been really great so far.
@smerchek, thank you so much for chiming in here. Their offering looks perfect actually. I'd love to see an example of this. An official example would be awesome and it'd help me a lot to visualize the changes that need to happen for something like this. If you or anyone else would have time to put something like that together that would be amazing.
Either way, I'm currently leaning in that direction.
What about hosting something open source like the Elastic Stack on fly yourself? All the big enterprise I have worked for self hosted the Elastic stack.
I saw this article related to Fly about it: https://fly.io/laravel-bytes/integrating-the-elastic-stack-elk-into-a-laravel-app-on-fly/
It seems like Fly uses the Elastic Stack themself as well: https://fly.io/blog/fly-behind-the-scenes-fresh-logging/
It is probably the cheapest way, and Kibana is really an extremely nice way to visualize incoming log data.
