Do we need Otel collector

[nascentcore-eng]

Starship includes a Prom collector, which can pull data from pods and services. Due to prometheus' data model, such pulling based approach works naturally, and push-based model (where service/pods proactively write data to prom collector) works best.

For Otel, its data are usually pushed from service and pods, because the detailed data is needed for the full value of the Observability.

Since we already have a PromScale connector sitting in between pods/services and timcescale db, it serves as some form of data aggregator.

• [ ] Investigate if PromScale has designs to support horizontal scaling of data sources.

A shared Otel collector mainly is for saving the needs for duplicate otel collectors configured for different applications, if users desire to do so.

• [ ] Research if users may prefer having OTel collector for applications, or there is concrete benefits of having OTel collectors.