jsonnet-libs
jsonnet-libs copied to clipboard
grafana
Apache HTTP codes histogram is not displayed correctly
I'm using Grafana Agent v0.33.2 with Apache HTTP integration v0.0.1 which sends metrics and logs to the Grafana Cloud instance.
My problem is that the "logs by HTTP codes" histogram does not correctly display the number of HTTP codes per bucket. It appears that almost all HTTP code buckets have roughly the same number of responses.
I think this happens because each histogram bucket also includes all codes below it. For example, in this case, there were actually 541 200-299 responses, three 300-399 responses, and three 400-499 responses. However, on the displayed histogram, all these numbers are summed together.
I'm not sure, but I think this might also affect the "errors rate" panel in the main dashboard and the "ApacheErrorsRateTooHigh" alert rule. Edit: After some very likely false positive "ApacheErrorsRateTooHigh" alerts, I'm pretty sure it also affects that.
Grafana Agent Config
integrations:
agent:
enabled: true
relabel_configs:
- action: replace
source_labels: [agent_hostname]
target_label: instance
metric_relabel_configs:
- source_labels: [exported_job]
target_label: job
- source_labels: [exported_instance]
target_label: instance
- action: labeldrop
regex: (exported_instance|exported_job)
node_exporter:
enabled: true
relabel_configs:
- target_label: instance
replacement: my-hostname
mysqld_exporter:
enabled: true
instance: my-hostname
data_source_name: "{{grafana_mysql_username}}:{{grafana_mysql_password}}@(localhost:3306)/"
relabel_configs:
- source_labels: [__address__]
target_label: job
replacement: integrations/mysql
apache_http:
enabled: true
instance: my-hostname
scrape_uri: "https://my-hostname/server-status?auto"
prometheus_remote_write:
- basic_auth:
password: {{grafana_api_key}}
username: {{grafana_metrics_id}}
url: {{grafana_metrics_url}}
logs:
configs:
- name: integrations
clients:
- basic_auth:
password: {{grafana_api_key}}
username: {{grafana_logs_id}}
url: {{grafana_logs_url}}
positions:
filename: /tmp/grafana-agent-positions.yaml
target_config:
sync_period: 10s
scrape_configs:
- job_name: integrations/agent
journal:
max_age: 24h
labels:
job: integrations/agent
instance: my-hostname
pipeline_stages:
- match:
action: drop
selector: '{unit!="grafana-agent.service"}'
drop_counter_reason: only_keep_grafana_agent_logs
- regex:
expression: '(level=(?P<log_level>[\s]*debug|warn|info|error))'
- labels:
level: log_level
relabel_configs:
- source_labels: [__journal__systemd_unit]
target_label: unit
- job_name: integrations/node_exporter_journal_scrape
journal:
max_age: 24h
labels:
job: integrations/node_exporter
instance: my-hostname
relabel_configs:
- source_labels: [__journal__systemd_unit]
target_label: unit
- source_labels: [__journal__boot_id]
target_label: boot_id
- source_labels: [__journal__transport]
target_label: transport
- source_labels: [__journal_priority_keyword]
target_label: level
- job_name: integrations/node_exporter_direct_scrape
static_configs:
- targets: [localhost]
labels:
__path__: /var/log/{syslog,messages,*.log}
job: integrations/node_exporter
instance: my-hostname
- job_name: integrations/mysql
static_configs:
- labels:
__path__: /var/log/mysql/*.log
job: integrations/mysql
instance: my-hostname
pipeline_stages:
- regex:
expression: '(?P<timestamp>.+) (?P<thread>[\d]+) \[(?P<label>.+?)\]( \[(?P<err_code>.+?)\] \[(?P<subsystem>.+?)\])? (?P<msg>.+)'
- labels:
label:
err_code:
subsystem:
- drop:
expression: "^ *$"
drop_counter_reason: "drop empty lines"
- job_name: integrations/apache_error
static_configs:
- targets: [localhost]
labels:
__path__: /var/log/apache2/error.log
job: integrations/apache_http
instance: my-hostname
pipeline_stages:
- regex:
# https://regex101.com/r/zNIq1V/1
expression: '^\[[^ ]* (?P<timestamp>[^\]]*)\] \[(?:(?P<module>[^:\]]+):)?(?P<level>[^\]]+)\](?: \[pid (?P<pid>[^\]]*)\])?(?: \[client (?P<client>[^\]]*)\])? (?P<message>.*)$'
- labels:
module:
level:
- static_labels:
logtype: error
- job_name: integrations/apache_access
static_configs:
- targets: [localhost]
labels:
__path__: /var/log/apache2/access.log
job: integrations/apache_http
instance: my-hostname
pipeline_stages:
- regex:
# https://regex101.com/r/9G75bY/1
expression: '^(?P<ip>[^ ]*) [^ ]* (?P<user>[^ ]*) \[(?P<timestamp>[^\]]*)\] "(?P<method>\S+)(?: +(?P<path>[^ ]*) +\S*)?" (?P<code>[^ ]*) (?P<size>[^ ]*)(?: "(?P<referer>[^\"]*)" "(?P<agent>.*)")?$'
- metrics:
response_http_codes:
type: Histogram
description: "Apache responses by HTTP codes"
prefix: apache_
source: code
config:
buckets: [199, 299, 399, 499, 599]
- labels:
method:
code:
- static_labels:
logtype: access
metrics:
configs:
- name: integrations
remote_write:
- basic_auth:
password: {{grafana_api_key}}
username: {{grafana_metrics_id}}
url: {{grafana_metrics_url}}
global:
scrape_interval: 60s
wal_directory: /tmp/grafana-agent-wal
Hi, I also have this issue and would be glad if someone can help. Kind regads,
