Enhance Prometheus plugin to handle multiple instances to proxy to

[Xantier]

Taken from: https://github.com/RoadieHQ/roadie-backstage-plugins/pull/230#discussion_r721435535

another question - can this be somehow overriden through annotations instead in the app-config? internally we have hundreds of prometheus servers deployed, through kubernetes namespaces. that would mean service A and service B send data to different servers (and the metrics are all aggregated by Thanos). assuming Thanos and Prometheus don't have a matching REST API Spec, as an integrator, I would like to be able to specify which path to query for each of my entities (or group of entities, somehow).

how could we model this?

Possible Implementation

We could do something like adding an additional annotation to the entity that can be used as part of proxy rewrite path. Something like prometheus.io/server: someservername as an annotation would call the API api/prometheus?server=someservername and the API implementation would use the same http-proxy-middleware as current proxy implementation, but with additional logic to construct the target URL based on passed in params.

[roadie-bot]

https://app.shortcut.com/larder/story/4306

[julioz]

@Xantier I am having a bit of trouble understanding how we could plug an annotation into the entities to replace the implementation of this method. Is there another plug-in, that you could point me to, that has a similar use-case?

I am not confident also that passing full URLs to the prometheus servers is the best idea (works, but...) to make this future-proof either. An entity would look like this:

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: sample-service-3
  description: |
    A service for testing Buildkite and Prometheus plugins
  annotations:
    buildkite.com/project-slug: exampleorganization/exampleproject
    prometheus.io/alert: 'all'
    prometheus.io/rule: node_memory_Active_bytes|instance,memUsage
+   prometheus.io/server: http://prometheus-operated.sample-service-3.myinternalnetwork.com:9090/

spec:
  type: service
  owner: roadie
  lifecycle: experimental
  providesApis:
    - sample-service-3

Maybe there is a better architecture for this - a backend component for the plugin that translates fields pulled from the entity itself (via useEntity()) into a URL, do you think that'd make sense? I am keen on following patterns from other existing plug-ins, if you can point me to any.

[Xantier]

Hi @julioz,

Thanks for taking a look at this. I think there are few options we could consider here.

1. Abusing the path rewrite in the proxy configuration to determine a URL based on params passed in. See how that could work.

   • The downside of this is that since all the path rewrite is able to do is rewrite the path. Only a single domain would work

2. Implementing a little bit more sophisticated proxying system than the current one is

   • This would mean to have the possibility to rewrite the domain/subdomain as well based on values passed in, on top of the path rewrite
   • See [1].

3. Implementing a backend specific to this plugin

   • This would be essentially the same as option 2 but not a generic implementation of is

I think the best option would likely be number 2. Number 3 should be simple enough to implement as a first solution but if we want to have a bit more generic option, number 2 isn't that far away from 3.

What do you think @julioz ?

[1] Enabling the possibility of configuration option like:

dynamicProxy:
    '/prometheus/api':
      target: https://${prometheusSubDomain}.internal-prom.com/api/v1/

Which then can be called similarly like so:

const subdomain = `someServiceNameOrSomethingThatWeCanIdentifyFromTheEntityAndWhichByConventionMatchTheSubdomain`
const proxyUrl = await this.discoveryApi.getBaseUrl('proxy');
fetch(`${proxyUrl}${this.proxyPath}?prometheusSubDomain=${subdomain}`);

[dweber019]

Why not using the same logic here https://github.com/RoadieHQ/roadie-backstage-plugins/tree/main/plugins/frontend/backstage-plugin-argo-cd#support-for-multiple-argocd-instances---option-1.

[Xantier]

Thanks @dweber019. That's a good idea and would work really well for this use case also. Getting the config file constructed for hundreds of Prometheus servers to match wanted might be a bit cumbersome but I think for the vast majority of use cases it would enable multiple Prometheus targets for this.

[julioz]

I think implementation-wise it could make sense, but if I understand the ArgoCD README correctly:

then in the following component definition annotations add a line with the url to the desired proxy path

this means entity authors would specify the proxy path as part of their catalog-info.yaml files. I don't like this solution very much because it couples entity descriptor files to the implementation of the Backstage instance. If the maintainers of the Backstage instance decide to change how they name their proxy paths, they would have to inform all entity authors to update their Entity annotations (or write a processor that does that).

What we have done in the Periskop plugin could be an alternative; defining instance names (in app-config.yaml) that then get used as keys in the entity descriptor files.

[Xantier]

I like the idea of instance names as descriptors, and then referring to those in catalog files. Exposing the config directly to the frontend in this case wouldn't be the ideal approach I think though. We'd want to expose the possibility to configure an auth header or other such more secretive items so the communications should go through the backend.

[dweber019]

Agree a level of indirection helps here. I would suggest something like this.

catalog-info.yaml

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: sample-service-3
  description: |
    A service for testing Buildkite and Prometheus plugins
  annotations:
    prometheus.io/alert: 'all'
    prometheus.io/rule: node_memory_Active_bytes|instance,memUsage 
    prometheus.io/service-name: prometheusTeamB #optional as there is a default at prometheus.proxyPath

app-config.yaml

proxy: 
  '/prometheus/api':
    target: http://localhost:9090/api/v1/
  '/prometheusTeamB/api':
    target: http://localhost:9999/api/v1/

prometheus:
   proxyPath: /prometheus/api
   instances:
     - name: prometheusTeamB
       proxyPath: /prometheusTeamB/api

[dweber019]

Created a PR with the above suggestion.

[Xantier]

Closed with https://github.com/RoadieHQ/roadie-backstage-plugins/pull/614