Warn when users specify custom metrics with no actual metric

[bchanan03]

Hi, Using PgAdmin connecting via user: root to the database and running the select works. Adding same SQL into the queries.yaml confimap and patch/deleting the deployment and recreating the deployment....the metrics is missing. Note: same user is used inside the deployment.

only metric which appears (out of the 3 queries): pg_postmaster metric missing: pg_index_usage_*

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-exporter-configmap
  labels:
    app: postgres-exporter
    monitoring: prometheus
data:
  queries.yaml  : |+
    pg_index_usage:
      query: "SELECT t.tablename AS relation, indexname, c.reltuples AS num_rows, pg_relation_size(quote_ident(t.tablename)::text) AS table_size, pg_relation_size(quote_ident(indexrelname)::text) AS index_size, idx_scan AS number_of_scans, idx_tup_read AS tuples_read, idx_tup_fetch AS tuples_fetched FROM pg_tables t LEFT OUTER JOIN pg_class c ON t.tablename=c.relname LEFT OUTER JOIN ( SELECT c.relname AS ctablename, ipg.relname AS indexname, x.indnatts AS number_of_columns, idx_scan, idx_tup_read, idx_tup_fetch, indexrelname, indisunique FROM pg_index x JOIN pg_class c ON c.oid = x.indrelid JOIN pg_class ipg ON ipg.oid = x.indexrelid JOIN pg_stat_all_indexes psai ON x.indexrelid = psai.indexrelid ) AS foo ON t.tablename = foo.ctablename WHERE t.schemaname='public' ORDER BY 1,2 "
      metrics:
        - relation:
            usage: "LABEL"
            description: "Relation name"
        - indexname:
            usage: "LABEL"
            description: "Index name"
        - num_rows:
            usage: "COUNTER"
            description: "Index number of rows"
        - table_size:
            usage: "COUNTER"
            description: "Table size"
        - index_size:
            usage: "COUNTER"
            description: "Index size"
        - number_of_scans:
            usage: "COUNTER"
            description: "Number of index scans"
        - tuples_read:
            usage: "COUNTER"
            description: "Number of tupels read"
        - tuples_fetched:
            usage: "COUNTER"
            description: "Number of tupels fetched"

    pg_postmaster:
      query: "SELECT pg_postmaster_start_time as start_time_seconds from pg_postmaster_start_time()"
      metrics:
        - start_time_seconds:
            usage: "GAUGE"
            description: "Time at which postmaster started"

    pg_stat_user_tables:
      query: "SELECT schemaname, relname, seq_scan, seq_tup_read, idx_scan, idx_tup_fetch, n_tup_ins, n_tup_upd, n_tup_del, n_tup_hot_upd, n_live_tup, n_dead_tup, n_mod_since_analyze, last_vacuum, last_autovacuum, last_analyze, last_autoanalyze, vacuum_count, autovacuum_count, analyze_count, autoanalyze_count FROM pg_stat_user_tables"
      metrics:
        - schemaname:
            usage: "LABEL"
            description: "Name of the schema that this table is in"
        - relname:
            usage: "LABEL"
            description: "Name of this table"
        - seq_scan:
            usage: "COUNTER"
            description: "Number of sequential scans initiated on this table"
        - seq_tup_read:
            usage: "COUNTER"
            description: "Number of live rows fetched by sequential scans"
        - idx_scan:
            usage: "COUNTER"
            description: "Number of index scans initiated on this table"
        - idx_tup_fetch:
            usage: "COUNTER"
            description: "Number of live rows fetched by index scans"
        - n_tup_ins:
            usage: "COUNTER"
            description: "Number of rows inserted"
        - n_tup_upd:
            usage: "COUNTER"
            description: "Number of rows updated"
        - n_tup_del:
            usage: "COUNTER"
            description: "Number of rows deleted"
        - n_tup_hot_upd:
            usage: "COUNTER"
            description: "Number of rows HOT updated (i.e., with no separate index update required)"
        - n_live_tup:
            usage: "GAUGE"
            description: "Estimated number of live rows"
        - n_dead_tup:
            usage: "GAUGE"
            description: "Estimated number of dead rows"
        - n_mod_since_analyze:
            usage: "GAUGE"
            description: "Estimated number of rows changed since last analyze"
        - last_vacuum:
            usage: "GAUGE"
            description: "Last time at which this table was manually vacuumed (not counting VACUUM FULL)"
        - last_autovacuum:
            usage: "GAUGE"
            description: "Last time at which this table was vacuumed by the autovacuum daemon"
        - last_analyze:
            usage: "GAUGE"
            description: "Last time at which this table was manually analyzed"
        - last_autoanalyze:
            usage: "GAUGE"
            description: "Last time at which this table was analyzed by the autovacuum daemon"
        - vacuum_count:
            usage: "COUNTER"
            description: "Number of times this table has been manually vacuumed (not counting VACUUM FULL)"
        - autovacuum_count:
            usage: "COUNTER"
            description: "Number of times this table has been vacuumed by the autovacuum daemon"
        - analyze_count:
            usage: "COUNTER"
            description: "Number of times this table has been manually analyzed"
        - autoanalyze_count:
            usage: "COUNTER"
            description: "Number of times this table has been analyzed by the autovacuum daemon"

[bchanan03]

debug logs

+ postgres-exporter-1051194761-22spg › postgres-exporter
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=info msg="Established new database connection." source="postgres_exporter.go:970"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="Querying Postgres Version" source="postgres_exporter.go:906"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=info msg="Semantic Version Changed: 0.0.0 -> 9.5.4" source="postgres_exporter.go:923"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="write_lag is being forced to discard due to version incompatibility." source="postgres_exporter.go:515"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="replay_lag is being forced to discard due to version incompatibility." source="postgres_exporter.go:515"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="pg_current_wal_lsn is being forced to discard due to version incompatibility." source="postgres_exporter.go:515"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="sent_lsn is being forced to discard due to version incompatibility." source="postgres_exporter.go:515"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="procpid is being forced to discard due to version incompatibility." source="postgres_exporter.go:515"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="write_lsn is being forced to discard due to version incompatibility." source="postgres_exporter.go:515"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="flush_lsn is being forced to discard due to version incompatibility." source="postgres_exporter.go:515"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="replay_lsn is being forced to discard due to version incompatibility." source="postgres_exporter.go:515"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="flush_lag is being forced to discard due to version incompatibility." source="postgres_exporter.go:515"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="pg_wal_lsn_diff is being forced to discard due to version incompatibility." source="postgres_exporter.go:515"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="New user metric namespace from YAML: pg_index_usage" source="postgres_exporter.go:415"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="New user metric namespace from YAML: pg_postmaster" source="postgres_exporter.go:415"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="New user metric namespace from YAML: pg_stat_user_tables" source="postgres_exporter.go:415"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="Adding new metric pg_stat_user_tables from user YAML file." source="postgres_exporter.go:474"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="Adding new metric pg_index_usage from user YAML file." source="postgres_exporter.go:474"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="Adding new metric pg_postmaster from user YAML file." source="postgres_exporter.go:474"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="Adding new query override pg_index_usage from user YAML file." source="postgres_exporter.go:485"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="Adding new query override pg_postmaster from user YAML file." source="postgres_exporter.go:485"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="Adding new query override pg_stat_user_tables from user YAML file." source="postgres_exporter.go:485"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="Querying pg_setting view" source="pg_setting.go:17"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="Querying namespace:  pg_index_usage" source="postgres_exporter.go:886"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="Querying namespace:  pg_postmaster" source="postgres_exporter.go:886"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="Querying namespace:  pg_stat_replication" source="postgres_exporter.go:886"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="Querying namespace:  pg_stat_bgwriter" source="postgres_exporter.go:886"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="Querying namespace:  pg_stat_database_conflicts" source="postgres_exporter.go:886"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="Querying namespace:  pg_locks" source="postgres_exporter.go:886"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="Querying namespace:  pg_stat_user_tables" source="postgres_exporter.go:886"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="Querying namespace:  pg_stat_activity" source="postgres_exporter.go:886"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=debug msg="Querying namespace:  pg_stat_database" source="postgres_exporter.go:886"
postgres-exporter-1051194761-22spg postgres-exporter time="2017-11-05T09:20:14Z" level=info msg="Starting Server: :9187" source="postgres_exporter.go:1093"

[bchanan03]

from pgAdmin (partial output).

device_catalog=> SELECT t.tablename AS relation, indexname, c.reltuples AS num_rows, pg_relation_size(quote_ident(t.tablename)::text) AS table_size, pg_relation_size(quote_ident(indexrelname)::text) AS index_size, idx_scan AS number_of_scans, idx_tup_read AS tuples_read, idx_tup_fetch AS tuples_fetched FROM pg_tables t LEFT OUTER JOIN pg_class c ON t.tablename=c.relname LEFT OUTER JOIN ( SELECT c.relname AS ctablename, ipg.relname AS indexname, x.indnatts AS number_of_columns, idx_scan, idx_tup_read, idx_tup_fetch, indexrelname, indisunique FROM pg_index x JOIN pg_class c ON c.oid = x.indrelid JOIN pg_class ipg ON ipg.oid = x.indexrelid JOIN pg_stat_all_indexes psai ON x.indexrelid = psai.indexrelid ) AS foo ON t.tablename = foo.ctablename WHERE t.schemaname='public' ORDER BY 1,2;
              relation              |                            indexname                            |  num_rows   | table_size | index_size | number_of_scans | tuples_read | tuples_fetched
------------------------------------+-----------------------------------------------------------------+-------------+------------+------------+-----------------+-------------+----------------
 auth_group                         | auth_group_name_a6ea08ec_like                                   |           0 |          0 |       8192 |               0 |           0 |              0
 auth_group                         | auth_group_name_key                                             |           0 |          0 |       8192 |               0 |           0 |              0
 auth_group                         | auth_group_pkey                                                 |           0 |          0 |       8192 |               0 |           0 |              0
 auth_group_permissions             | auth_group_permissions_0e939a4f                                 |           0 |          0 |       8192 |               0 |           0 |              0
 auth_group_permissions             | auth_group_permissions_8373b171                                 |           0 |          0 |       8192 |               0 |           0 |              0
 auth_group_permissions             | auth_group_permissions_group_id_0cd325b0_uniq                   |           0 |          0 |       8192 |               0 |           0 |              0
 auth_group_permissions             | auth_group_permissions_pkey                                     |           0 |          0 |       8192 |               0 |           0 |              0
 auth_permission                    | auth_permission_417f1b1c                                        |          48 |       8192 |      16384 |               0 |           0 |              0
 auth_permission                    | auth_permission_content_type_id_01ab375a_uniq                   |          48 |       8192 |      16384 |               0 |           0 |              0
 auth_permission                    | auth_permission_pkey                                            |          48 |       8192 |      16384 |               0 |           0 |              0
 auth_user                          | auth_user_pkey                                                  |           0 |          0 |       8192 |               0 |           0 |              0
 auth_user                          | auth_user_username_6821ab7c_like                                |           0 |          0 |       8192 |               0 |           0 |              0
 auth_user                          | auth_user_username_key                                          |           0 |          0 |       8192 |               0 |           0 |              0
 auth_user_groups                   | auth_user_groups_0e939a4f                                       |           0 |          0 |       8192 |               0 |           0 |              0
 auth_user_groups                   | auth_user_groups_e8701ad4                                       |           0 |          0 |       8192 |               0 |           0 |              0
 auth_user_groups                   | auth_user_groups_pkey                                           |           0 |          0 |       8192 |               0 |           0 |              0
 auth_user_groups                   | auth_user_groups_user_id_94350c0c_uniq                          |           0 |          0 |       8192 |               0 |           0 |              0

[wrouesnel]

Hm this is an interesting one. The queries.yaml code hasn't had as much testing as I would like at the moment (see self raised issues about it) - but it's on agenda to revamp for a release.

When you say a root user...do you mean the postgres user?

[bchanan03]

yes. here is yet another example:

    pg_database_size:
      query: "select d.datname as table_name, pg_catalog.pg_get_userbyid(d.datdba) as table_owner, case when pg_catalog.has_database_privilege(d.datname, 'connect') then pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname)) else 'no access' end as table_size from pg_catalog.pg_database d order by case when pg_catalog.has_database_privilege(d.datname, 'connect') then pg_catalog.pg_database_size(d.datname) else null end"
      metrics:
        - table_name:
            usage: "LABEL"
            description: "Table name"
        - table_owner:
            usage: "LABEL"
            description: "Table Owner name"
        - table_size:
            usage: "LABEL"
            description: "Table size"

again i was able to run the SQL into pgAdmin (deprecating some of our env tables)

postgres=> SELECT d.datname AS Name, pg_catalog.pg_get_userbyid(d.datdba) AS Owner, CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT') THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname)) ELSE 'No Access' END AS SIZE FROM pg_catalog.pg_database d ORDER BY CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT') THEN pg_catalog.pg_database_size(d.datname) ELSE NULL END;
         name         |        owner         |   size
----------------------+----------------------+-----------
 template0            | rdsadmin             | 7153 kB
 template1            | root                 | 7153 kB
 device_catalog       | device_catalog       | 16 GB
 rdsadmin             | rdsadmin             | No Access
 apigwdb              | apigw                | No Access
(18 rows)

Debug log enabled (notice: pg_database_size) - checking port 9187 (port-forward) metrics pg_database_size_table_name, pg_database_size_table_owner and pg_database_size_table_size are missing

+ postgres-exporter-1051194761-kllpx › postgres-exporter
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=info msg="Established new database connection." source="postgres_exporter.go:970"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="Querying Postgres Version" source="postgres_exporter.go:906"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=info msg="Semantic Version Changed: 0.0.0 -> 9.5.4" source="postgres_exporter.go:923"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="procpid is being forced to discard due to version incompatibility." source="postgres_exporter.go:515"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="sent_lsn is being forced to discard due to version incompatibility." source="postgres_exporter.go:515"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="flush_lag is being forced to discard due to version incompatibility." source="postgres_exporter.go:515"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="write_lsn is being forced to discard due to version incompatibility." source="postgres_exporter.go:515"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="pg_wal_lsn_diff is being forced to discard due to version incompatibility." source="postgres_exporter.go:515"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="write_lag is being forced to discard due to version incompatibility." source="postgres_exporter.go:515"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="flush_lsn is being forced to discard due to version incompatibility." source="postgres_exporter.go:515"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="replay_lag is being forced to discard due to version incompatibility." source="postgres_exporter.go:515"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="replay_lsn is being forced to discard due to version incompatibility." source="postgres_exporter.go:515"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="pg_current_wal_lsn is being forced to discard due to version incompatibility." source="postgres_exporter.go:515"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="New user metric namespace from YAML: pg_database_size" source="postgres_exporter.go:415"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="New user metric namespace from YAML: pg_postmaster" source="postgres_exporter.go:415"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="New user metric namespace from YAML: pg_stat_user_tables" source="postgres_exporter.go:415"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="Adding new metric pg_stat_user_tables from user YAML file." source="postgres_exporter.go:474"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="Adding new metric pg_database_size from user YAML file." source="postgres_exporter.go:474"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="Adding new metric pg_postmaster from user YAML file." source="postgres_exporter.go:474"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="Adding new query override pg_database_size from user YAML file." source="postgres_exporter.go:485"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="Adding new query override pg_postmaster from user YAML file." source="postgres_exporter.go:485"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="Adding new query override pg_stat_user_tables from user YAML file." source="postgres_exporter.go:485"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="Querying pg_setting view" source="pg_setting.go:17"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="Querying namespace:  pg_stat_replication" source="postgres_exporter.go:886"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="Querying namespace:  pg_database_size" source="postgres_exporter.go:886"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="Querying namespace:  pg_stat_database" source="postgres_exporter.go:886"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="Querying namespace:  pg_locks" source="postgres_exporter.go:886"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="Querying namespace:  pg_stat_database_conflicts" source="postgres_exporter.go:886"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="Querying namespace:  pg_stat_user_tables" source="postgres_exporter.go:886"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="Querying namespace:  pg_postmaster" source="postgres_exporter.go:886"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="Querying namespace:  pg_stat_activity" source="postgres_exporter.go:886"
postgres-exporter-1051194761-kllpx postgres-exporter time="2017-11-05T15:09:25Z" level=debug msg="Querying namespace:  pg_stat_bgwriter" source="postgres_exporter.go:886"

[wrouesnel]

@bchanan03 sorry to take so long to get back to this (a broken shower means I'm not in my apartment which has severely messed with my side project time) - but the problem you're having with the pg_database_size metric there is all the columns are marked as LABEL types - you haven't actually defined anything to collect as a metric.

Change table_size to be a GAUGE and it should appear.

We really need a warning when a metric would be uncollectable due to something like this.

[CoolCold]

@bchanan03 can you please share the final queries yaml for table/index sizes ? So others (including me ;) ) won't reinvent the wheel

[dheeruymv]

Hi @wrouesnel : I'm facing exact same problem. Here is my query in queries.yml, I'm not able to solve, this executes completely fine on postgres.

As I see, postgres exporter needs a GUAGE,COUNTER or HISTOGRAM, I've put one column/metric as GUAGE, but still it doesn't appear in the /metrics end point.

I don't see anything error/exception/warning in debug logs as well. I've spent almost half a day on this! I couldn't get through, it works absolutely fine in postgres but not through postgres_exporter

Am i using this correctly? can you please help on this?

pgss_stat_act: query: "select pg_stat_activity.usename as username, pg_stat_activity.application_name as appname, pg_stat_activity.pid as processid, now()-pg_stat_activity.query_start as duration FROM pg_stat_activity" metrics:

• username: usage: "LABEL" description: "Username"
• appname: usage: "LABEL" description: "Application Name"
• processid: usage: "LABEL" description: "Process ID"
• duration: usage: "GUAGE" description: "Duration"