otel: simple OTEL collector/Prometheus stack for testing purposes

[krissetto]

- What I did Added a compose stack to be used for testing out OTEL functionality, mostly based on our current documentation PR found here.

- How I did it

• Added compose stack;
• Added basic OTEL collector and prometheus configs;
• Added basic usage doc

- How to verify it Read the doc and try it out :)

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

[thaJeztah]

Slightly wondering if we need the scripts at all, as they (effectively) are just a docker compose up?

(I know we have a bunch of ahack/ script already though, but mostly looking if for this one we need more)

[laurazard]

IMO a big part of why I want this change is to make local development easier, so Testcontainers don't really achieve that. I'm okay with shell scripts are they're portable and easy to work with and don't require bringing in additional dependencies, but if atm they are really just doing a compose up, compose down, etc., then I'm okay with holding off on them and just adding the Compose file, and thinking about the rest later if we feel the need for them.

[thaJeztah]

Yes, agreed on the "local development" part. It's usually also my reservation on things like mage as replacement for make; it's cool as it's in Go (and can be flexible), but also means that go is now a requirement on your local machine (and where possible, we should try to still allow for a dockerized workflow, so that there's no real requirements other than "have docker installed")

[krissetto]

I'm not super fond of the scripts either, I was trying to follow a bit of what already existed while keeping things as simple s possible for the community (eg. i think buildx has a script which completely wraps compose, for example, and i thought that was a bit too much).
If we agree on removing them completely, I'll adjust the usage.md to reflect that.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 61.05%. Comparing base (b982833) to head (c91c271). Report is 28 commits behind head on master.

:exclamation: Current head c91c271 differs from pull request most recent head e1dcc19. Consider uploading reports for the commit e1dcc19 to get more accurate results

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #5026   +/-   ##
=======================================
  Coverage   61.05%   61.05%           
=======================================
  Files         295      295           
  Lines       20647    20647           
=======================================
  Hits        12605    12605           
  Misses       7146     7146           
  Partials      896      896           

[krissetto]

ps i added aspire dashboard here as well so we can test both and see what we want to stick with