opentelemetry-operator
opentelemetry-operator copied to clipboard
open-telemetry
Use `/job` endpoint instead of generated promconfig
Currently the operator uses jobs generated by the promconfig to extract a list of relevant jobs from the target allocator.
This is working as long as all target updates are generated by the operator reconciliation, but this prevents the implementation of other sources in the TargetAllocator, e.g. custom implementation of the TargetAllocatorAPI or PrometheusCR.
To resolve this following solutions have been proposed:
- Call the /jobs endpoint from the Operator. This is not ideal as this would introduce a requirement that the TargetAllocator has to be reachable by the operator. I think this would be the most flexible solution regarding the addition of new watcher in the TargetAllocator or BYO target allocators.
- Add a catch all job for PrometheusCR jobs if this feature is enabled. In that case the metrics generated would not be the same as the ones by the PrometheusOperator, as the job label would differ between the two implementations.
- Move the PrometheusCRD ( ServiceMonitor and PodMonitor ) watcher into the operator and only provide the resolved configuration to the TargetAllocator This can lead to high load on big clusters, as the operator would watch the CRDs for all Collector instances
- Create an Observer for the TargetAllocator and extend the ReceiverCreator to allow looping over observer results. This seems to be cleanest solution IMO.
Further @Aneurysm9 has proposed to add the functionality to retrieve the jobs directly in prometheusreceiver in this PR.
What about having the prometheus receiver watch the /jobs endpoint and update its configuration as appropriate. This could be managed by a config flag to limit impact where the target allocator is not in use and would not require creating new receivers or reloading pipelines.
This IMO is the cleanest solutions, as it would not add any constrains and promises to be most efficient.
Should this go in collector-contrib? If we choose to handle this in the receiver then I don't think there will be any changes required in the operator.
I think there should be one here as well at the collector, as the problem exist for the operator and the solution has to be implemented in the collector.
If you are against this, I'm going to close this one and open a similar issue in collector-contrib repo.
I think there should be one here as well at the collector, as the problem exist for the operator and the solution has to be implemented in the collector.
Sure, I guess it makes sense to keep a tracking issue here.