apm-agent-python
apm-agent-python copied to clipboard
elastic
[META 693] Send logs through APM Agents
See meta issue and spec for the description and details:
- Meta issue: https://github.com/elastic/apm/issues/693
- Spec issue: https://github.com/elastic/apm/issues/694
Hello, any update on this? Is the only way to ingest logs into elastic to install the Elastic Agent on the hosts?
@gianfrancodemarco Yes, currently Elastic Agent or Filebeat (which is what the Elastic Agent uses) are your best options. We do plan to implement log sending from the python agent, but (1) I don't know when, and (2) it will always be slower than collecting it with Filebeat.
@basepi thank you very much, very clear. I have another related question, i don't know if it is OT: do you know if, after collecting logs in other ways e.g. via Amazon CloudWatch, it is possible/easy to use the log correlation and similar features?
It mostly depends on the intake. If you're using ecs-logging, and then intake with the Elastic Agent/Filebeat, then it should "just work". The key is you need the JSON to be ingested into Elasticsearch so those correlation fields (transaction.id, service.name, etc) are present in the log documents. I don't know the current status of our "ingesting from CloudWatch" story -- but I assume if there are ecs-formatted logs in CloudWatch our integration should handle them correctly.
Hello @basepi, just a nitpick. We allow log sending with the Flask framework https://www.elastic.co/guide/en/apm/agent/python/current/flask-support.html#flask-logging but I would recommend we write somewhere it's an experimental feature (as the Java one https://www.elastic.co/guide/en/apm/agent/java/current/logs.html).
@lucabelluccini Yes, I've been meaning to disable that "feature" because it's very inefficient. But I need to wait for a major version since it will be a breaking change.
This issue is to re-introduce log sending from the APM agent, but it needs to be written very differently to be efficient, and it needs to include more than Flask.