open-api icon indicating copy to clipboard operation
open-api copied to clipboard

Published 20 hours ago • verified publisher icon freeCodeCamp

Configure logging

Open ojongerius opened this issue 7 years ago • 4 comments

Where do our logs go, what do other freeCodeCamp project do? Can and if so, are we happy to re-use this?

/cc @freeCodeCamp/open-api

ojongerius avatar Apr 30 '18 04:04 ojongerius

Currently none of the projects have a persistence for logs, sure we do have some tools like opbeat and datadog but, we would want to have our own solution preferably cheap (as in free) like

https://www.elastic.co/solutions/metrics https://www.elastic.co/solutions/apm https://www.elastic.co/solutions/logging

We are exploring options for hosting them on our own.

raisedadead avatar Apr 30 '18 08:04 raisedadead

Thanks!

Is Datadog giving us a good deal, the lack of data retention over 24 hours in the free tier sounds like a blocker. As Opbeat has been acquired by Elastic thats a dead end.

The Elastic offerings sound interesting. I have seen quite a high operational load on keeping an ELK stack performing at scale, which makes me a little wary of using it in a small team. However if we get enough AWS credits to run the Elastic products for free it would definitely be an interesting option to explore!

How far are you in exploring running a self hosted ELK? Shall we set up a call this week and catch up?

ojongerius avatar Apr 30 '18 09:04 ojongerius

I am planning to re-route the beta instance logs to a small EC2 instance on AWS as a spike with the elastic stack. I am unsure when I will get that ready though. It will need some changes on the beta itself.

But yeah I am up for a call around Thursday if you would like to catch up. Will send you an invite.

raisedadead avatar Apr 30 '18 09:04 raisedadead

We've had our meeting and decided to use Cloudwatch for now. What remains to do for this issue is to improve log messages by adding the API GW ID

Lambda see https://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-logging.html Interesting trick to be able to track which API GW request caused a Lambda log message https://aws.amazon.com/blogs/compute/techniques-and-tools-for-better-serverless-api-logging-with-amazon-api-gateway-and-aws-lambda/ API gw see https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html Usage: https://docs.aws.amazon.com/lambda/latest/dg/monitoring-functions-logs.html

ojongerius avatar May 02 '18 23:05 ojongerius