[9.1](backport #48020) feat: add `add_cloud_metadata` processor to OTel Beat processor
Proposed commit message
Add add_cloud_metadata processor to the OTel Beat processor.
Checklist
- [x] My code follows the style guidelines of this project
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- ~~[ ] I have made corresponding change to the default configuration files~~
- [x] I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the
stresstest.shscript to run them under stress conditions and race detector to verify their stability. - ~~[ ] I have added an entry in
./changelog/fragmentsusing the changelog tool.~~
How to test this PR locally
- Include the processor in Elastic Agent by adding it to https://github.com/elastic/elastic-agent/blob/89e0e9a5b0ebb376773de813a1fd2c0b2f30e74a/internal/pkg/otel/components.go.
- Build the Elastic Agent binary
- Run the Agent on a cloud instance, e.g. an AWS EC2 instance, with an OTel config that includes the Beat processor.
Example OTel config:
service:
pipelines:
logs:
receivers:
- filebeatreceiver
processors:
- beat
receivers:
filebeatreceiver:
filebeat:
inputs:
- type: filestream
id: logs-from-files
paths:
- /var/log/*.log
processors: []
output:
otelconsumer:
path.data: /tmp/1210/data
path.logs: /tmp/1210/logs
queue.mem:
flush.timeout: 0
processors:
beat:
processors:
- add_cloud_metadata:
exporters:
debug:
sending_queue:
enabled: false
use_internal_logger: false
verbosity: detailed
```<hr>This is an automatic backport of pull request #48020 done by [Mergify](https://mergify.com).
Cherry-pick of e1630bed1fa73f3caaecf8945e2990c3158112bb has failed:
On branch mergify/bp/9.1/pr-48020
Your branch is up to date with 'origin/9.1'.
You are currently cherry-picking commit e1630bed1.
(fix conflicts and run "git cherry-pick --continue")
(use "git cherry-pick --skip" to skip this patch)
(use "git cherry-pick --abort" to cancel the cherry-pick operation)
Changes to be committed:
modified: x-pack/otel/processor/beatprocessor/README.md
modified: x-pack/otel/processor/beatprocessor/processor.go
modified: x-pack/otel/processor/beatprocessor/processor_test.go
Unmerged paths:
(use "git add <file>..." to mark resolution)
both modified: x-pack/filebeat/tests/integration/otel_test.go
To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally
:robot: GitHub comments
Just comment with:
rundocs-build: Re-trigger the docs validation. (use unformatted text in the comment!)
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)
This pull request has not been merged yet. Could you please review and merge it @andrzej-stencel? 🙏