BUG: Agent ingest of graphite plaintext metrics does not ingest graphite tags.

[jspaleta]

Agent ingest of graphite plaintext metrics does not ingest graphite tags.

Expected Behavior

tags should be extracted and converted to Sensu metrics tags Sensu metric name should not include graphite tag information (starting with semicolon)

Current Behavior

if graphite plaintext has tags, tags are encoded as part of the sensu metric name

Possible Solution

Fix agent to identify tagging format: https://graphite.readthedocs.io/en/latest/tags.html#graphite-tag-support

Steps to Reproduce (for bugs)

1. Create check that outputs graphite metrics with tags:

os.disk.used_percent;filesystem=/etc/resolv.conf 69.5 1647967211
os.disk.used_bytes;filesystem=/etc/resolv.conf 41368014848 1647967211
os.disk.free_bytes;filesystem=/etc/resolv.conf 18140893184 1647967211
os.disk.total_bytes;filesystem=/etc/resolv.conf 62725623808 1647967211
os.disk.used_percent;filesystem=/etc/hostname 69.5 1647967211
os.disk.used_bytes;filesystem=/etc/hostname 41368014848 1647967211
os.disk.free_bytes;filesystem=/etc/hostname 18140893184 1647967211
os.disk.total_bytes;filesystem=/etc/hostname 62725623808 1647967211
os.disk.used_percent;filesystem=/etc/hosts 69.5 1647967211
os.disk.used_bytes;filesystem=/etc/hosts 41368014848 1647967211
os.disk.free_bytes;filesystem=/etc/hosts 18140893184 1647967211
os.disk.total_bytes;filesystem=/etc/hosts 62725623808 1647967211
os.disk.used_percent;filesystem=/var/lib/sensu 69.5 1647967211
os.disk.used_bytes;filesystem=/var/lib/sensu 41368014848 1647967211
os.disk.free_bytes;filesystem=/var/lib/sensu 18140893184 1647967211
os.disk.total_bytes;filesystem=/var/lib/sensu 62725623808 1647967211

type: CheckConfig
api_version: core/v2
metadata:
  created_by: admin
  name: graphite_tag_test
  namespace: default
spec:
  check_hooks: null
  command: cat /tmp/graphite_test.txt
  env_vars: null
  handlers:
  - diagnostic
  high_flap_threshold: 0
  interval: 60
  low_flap_threshold: 0
  output_metric_format: graphite_plaintext
  output_metric_handlers: null
  pipelines: []
  proxy_entity_name: ""
  publish: false
  round_robin: false
  runtime_assets: null
  secrets: null
  stdin: false
  subdue: null
  subscriptions:
  - test
  timeout: 10
  ttl: 0

2. Make sure you have a diagnostic handler that can output event metrics

type: Handler
api_version: core/v2
metadata:
  created_by: admin
  labels:
    sensu.io/managed_by: sensuctl
  name: diagnostic
  namespace: default
spec:
  command: jq . > /tmp/event.json
  env_vars: null
  filters: null
  handlers: null
  timeout: 0
  type: pipe

3. sensuctl check execute graphite_tag_test
4. Look at metrics field in /tmp/event.json produced by diagnostic handler. Notice no tags were generated, and metric names include tag information (starting with semicolon)

 "metrics": {
    "handlers": null,
    "points": [
      {
        "name": "os.disk.used_percent;filesystem=/etc/resolv.conf",
        "value": 69.5,
        "timestamp": 1647967211,
        "tags": null
      },
      {
        "name": "os.disk.used_bytes;filesystem=/etc/resolv.conf",
        "value": 41368014848,
        "timestamp": 1647967211,
        "tags": null
      },
      {
        "name": "os.disk.free_bytes;filesystem=/etc/resolv.conf",
        "value": 18140893184,
        "timestamp": 1647967211,
        "tags": null
      },
      {
        "name": "os.disk.total_bytes;filesystem=/etc/resolv.conf",
        "value": 62725623808,
        "timestamp": 1647967211,
        "tags": null
      },
      {
        "name": "os.disk.used_percent;filesystem=/etc/hostname",
        "value": 69.5,
        "timestamp": 1647967211,
        "tags": null
      },
      {
        "name": "os.disk.used_bytes;filesystem=/etc/hostname",
        "value": 41368014848,
        "timestamp": 1647967211,
        "tags": null
      },
      {
        "name": "os.disk.free_bytes;filesystem=/etc/hostname",
        "value": 18140893184,
        "timestamp": 1647967211,
        "tags": null
      },
      {
        "name": "os.disk.total_bytes;filesystem=/etc/hostname",
        "value": 62725623808,
        "timestamp": 1647967211,
        "tags": null
      },
      {
        "name": "os.disk.used_percent;filesystem=/etc/hosts",
        "value": 69.5,
        "timestamp": 1647967211,
        "tags": null
      },
      {
        "name": "os.disk.used_bytes;filesystem=/etc/hosts",
        "value": 41368014848,
        "timestamp": 1647967211,
        "tags": null
      },
      {
        "name": "os.disk.free_bytes;filesystem=/etc/hosts",
        "value": 18140893184,
        "timestamp": 1647967211,
        "tags": null
      },
      {
        "name": "os.disk.total_bytes;filesystem=/etc/hosts",
        "value": 62725623808,
        "timestamp": 1647967211,
        "tags": null
      },
      {
        "name": "os.disk.used_percent;filesystem=/var/lib/sensu",
        "value": 69.5,
        "timestamp": 1647967211,
        "tags": null
      },
      {
        "name": "os.disk.used_bytes;filesystem=/var/lib/sensu",
        "value": 41368014848,
        "timestamp": 1647967211,
        "tags": null
      },
      {
        "name": "os.disk.free_bytes;filesystem=/var/lib/sensu",
        "value": 18140893184,
        "timestamp": 1647967211,
        "tags": null
      },
      {
        "name": "os.disk.total_bytes;filesystem=/var/lib/sensu",
        "value": 62725623808,
        "timestamp": 1647967211,
        "tags": null
      }
    ]
  }

5. cry

Context

Follow up on custom support ticket (see internal ref https://secure.helpscout.net/conversation/1823538483/28609?folderId=1211661) . Customer trying to ingest graphite plaintext with tags.

Your Environment

$ sensuctl version sensuctl version 6.6.6+ee, enterprise edition, build adcbb0b56123cc13774cb951cd697efef636846b, built 2022-02-17T00:57:42Z, built with go1.17.1 $ /usr/sbin/sensu-backend version sensu-backend version 6.6.6+ee, enterprise edition, build adcbb0b56123cc13774cb951cd697efef636846b, built 2022-02-17T00:57:42Z, built with go1.17.1 $ /usr/sbin/sensu-agent version sensu-agent version 6.6.6+ee, enterprise edition, build adcbb0b56123cc13774cb951cd697efef636846b, built 2022-02-17T00:57:42Z, built with go1.17.1

[thoward]

Note, I did some code spelunking and it looks like the agent doesn't support parsing tags from graphite plain text at this time: https://github.com/sensu/sensu-go/blob/main/agent/transformers/graphite.go#L45 ... so this would be a feature add, vs bug fix.