[META 693] Send logs through APM Agents

[joshuaquek]

See meta issue and spec for the description and details:

• Meta issue: https://github.com/elastic/apm/issues/693
• Spec issue: https://github.com/elastic/apm/issues/694

---

Describe the bug

I am trying to do a simple NodeJS proof-of-concept app for a customer to showcase the NodeJS APM on Elastic Cloud, I have tried setting it up following the guide - https://www.elastic.co/guide/en/apm/agent/nodejs/current/express.html

I am able to get metrics but no logs are flowing in:

Also, I have included my code too as well the command line terminal, set to debug mode ...I created a dummy "Error" but this is not being captured in the logs or metrics as well:

Included the package.json file too as well as the live output on the actual libraries being used, in case it helps to shed more light on the issue:

The customer has tried also and is experiencing the same issue with their own proof-of-concept app on their end :disappointed: Would anyone happen to have any guidance on making this work? (i.e. Logs are not being sent back / no logs data received)

To Reproduce See above screenshots

Expected behaviour It should report logs back but it is not reporting any logs back. On the bright side, metrics are being reported back.

Environment (please complete the following information)

• OS: MacOS M1 Montery
• Node.js version: v18.6.0
• APM Server version: v8.3.3
• Agent version: v3.38.0

How are you starting the agent? (please tick one of the boxes) (Please see the screenshot provided)

• [✅ ] Calling agent.start() directly (e.g. require('elastic-apm-node').start(...))
• [ ] Requiring elastic-apm-node/start from within the source code
• [ ] Starting node with -r elastic-apm-node/start

Additional context See above screenshots

[joshuaquek]

Some good news - I have published a nodejs library called elastic-apm-node-logger (links below) that works hand-in-hand with our existing elastic-apm-node NodeJS APM Agent library that bridges this gap haha

...with this, our customers no longer need to install a separate Filebeat (i.e. Golang process) running alongside their existing NodeJS app process

Including the NPM link and Github source code links here: • Installation URL - https://www.npmjs.com/package/elastic-apm-node-logger • Source Code - https://github.com/joshuaquek/elastic-apm-node-logger

This is currently only a temporary stopgap solution, and I hope that our engineering team can build this feature into our official elastic-apm-node npm package here.

Cheers team!

[astorm]

Hey @joshuaquek, that looks like a pretty neat package! I've marked this as a feature request.

For folks reading along ingesting logs is not currently a feature of the agent -- instructions for getting your Node.js logs into your Elastic instance can be found here.

[joshuaquek]

Hi @astorm , thank you! Also, allow me to look at the ECS formatter link you presented - I can also see if there is a way to integrate this into the elastic-apm-node-logger package.

However, I am sure our engineering team will be able to build this whole logs data sending feature into our official elastic-apm-node agent and I can then deprecate the elastic-apm-node-logger NPM Library after that.

[BlueSialia]

Is this still a pending feature?