inspectit-ocelot icon indicating copy to clipboard operation
inspectit-ocelot copied to clipboard

Wrong metrics referral causes inspectit not to expose any metric on prometheus metrics endpoint

Open albersma69 opened this issue 3 years ago • 1 comments

The following example contains an error when refering to the metrics "pets". It references the view "pets/count" instead of the metric "pets" itself.

inspectit:
  metrics:
    definitions:
      '[pets]':
        type: LONG
        unit: count
        description: "The number of saves per pet type"
        views:
          '[pets/count]':
            aggregation: COUNT
            tags:
              pet_type : true
  instrumentation:
    scopes:
      's_save_pet':
        type: 
          name: 'org.springframework.samples.petclinic.customers.web.PetService'
        methods:
          - name: save
            matcher-mode: EQUALS_FULLY
    rules:
      'r_extract_pet_type':
        scopes:
          's_save_pet': true
        metrics:
          '[pets/count]':
            value: 1
            data-tags:
              pet_type: 'extract-pet-type'
        exit:
          extract-pet-type:
            action: 'a_extract_pet_type_from_save'

This leads to the following error message during startup of the instrumented java app: 021-11-17 12:41:54,155 ERROR 8635 --- [inspectIT] [ Thread-0] r.i.o.core.config.InspectitEnvironment : Error loading the configuration 'inspectit'. 2021-11-17 12:41:54,156 ERROR 8636 --- [inspectIT] [ Thread-0] r.i.o.core.config.InspectitEnvironment : instrumentation.rules.r_extract_pet_type.metrics => Metric 'pets/count' is not defined in inspectit.metrics.definitions! 2021-11-17 12:41:54,191 ERROR 8671 --- [inspectIT] [ Thread-0] r.i.o.core.config.InspectitEnvironment : Startup configuration is not valid! Using fallback configuration but listening for configuration updates...

Furthermore, the prometheus endpoint is dead, ie. not available. I'd expect that the this wrong config should be skipped and inspectIT would go on with the remaining instrumentation and keep surving the /metrics port. Even the default port 8888 is not served at all.

albersma69 avatar Nov 17 '21 13:11 albersma69

Hi @albersma69

yes, that's right. In the rule, you have to define the metric for which data should be exported and not its views. The views represents a "view" on a metric based on certain attributes, thus, the metric itself is responsible for collecting the data. Due to this, the rule must use the metric's name [pets] instead of one of its views.

    rules:
      'r_extract_pet_type':
        scopes:
          ...
        metrics:
          '[pets]':
            ...

Regarding the error: yes, that is the intended behavior of the agent. In case the configuration is invalid, it is rejected in order to prevent that data is recorded which my be invalid due to a broken or partial configuration. In this case, the last valid configuration will be used instead. In case this happens at startup, the agent falls back to a minimal configuration. That is also the reason that the Prometheus exporter is not started.

mariusoe avatar Nov 19 '21 09:11 mariusoe

Too me this reads like works as designed. Additionally, there was no further feedback in several months. Hence, we close this issue.

quandor avatar Mar 20 '23 08:03 quandor