Support chart labels in alerts

[MrZammler]

Summary

This PR allows (at this point) info fields for alerts to have variables populated from chart labels. To do this the user only needs to add a $label:chart_label_key entry in an info field. The variable will be replaced with the value of chart_label_key. The variable needs to begin with $lbl_.

In later stages, if new items are added to alert configs, we can use this to display more rich subjects on notifications, etc.

Test Plan

Pick an alert that is on a chart that has chart labels. Edit the info field and add a variable like described. For example, if the httpcheck_netdata.cloud.request_status chart has a label with key target:

Info field in alert config: info: ratio of failed requests either due to timeouts or no connection over the last 5 minutes for $label:target will become: ratio of failed requests either due to timeouts or no connection over the last 5 minutes for https://netdata.cloud/

Additional Information

For users: How does this change affect me?

[thiagoftsm]

Hello @MrZammler ,

Considering that I required changes in your PR, please, also rebase it with current master.

Best regards!

[MrZammler]

Thanks a lot Thiago!

[thiagoftsm]

Hello @MrZammler ,

When your PR is ready again, please, request a review from me and @ilyam8. :handshake:

Best regards!

[thiagoftsm]

I updated the alert:

 alarm: disk_inode_testing
    on: disk_inodes._mnt_vm
    os: linux
lookup: sum -10s foreach *
 units: byte
 every: 10s
  info: Show me $label:fstype
  warn: $this > 100
  crit: $this > 400

and I got the expected result:

The PR is working, after requests to be addressed I will approve it.

[ilyam8]

I mentioned it, but let me bring it up again - I think we need to change $lbl_ to something else (not use _ as a delimiter between the function and the label name). @MrZammler do you have any suggestions?

Just to name a few:

• using space: $lbl label_name or $lbl_get label_name.
• like function call: $lbl(label_name) or $lbl_get(label_name).
• colon: $lbl:label_name or $lbl_get:label_name.

[MrZammler]

I mentioned it, but let me bring it up again - I think we need to change $lbl_ to something else (not use _ as a delimiter between the function and the label name). @MrZammler do you have any suggestions?

Just to name a few:

* using space: `$lbl label_name` or `$lbl_get label_name`.

* like function call: `$lbl(label_name)` or `$lbl_get(label_name)`.

* colon: `$lbl:label_name` or `$lbl_get:label_name`.

Hi @ilyam8. Yes, thanks, it now looks for $label:label_name. Sorry I didn't mention it before.

[vlvkobal]

Please look at the Valgring report on Netdata exit

==63442== 148 bytes in 2 blocks are definitely lost in loss record 2,397 of 2,707
==63442==    at 0x4842888: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==63442==    by 0x56E767E: strdup (strdup.c:42)
==63442==    by 0x14A721: strdupz (libnetdata.c:159)
==63442==    by 0x191628: rrdcalc_replace_variables (rrdcalc.c:44)
==63442==    by 0x1919D6: rrdsetcalc_link (rrdcalc.c:143)
==63442==    by 0x192641: rrdcalc_add_to_host (rrdcalc.c:420)
==63442==    by 0x192A30: rrdcalc_create_from_template (rrdcalc.c:541)
==63442==    by 0x1939BB: rrdcalctemplate_check_conditions_and_link (rrdcalctemplate.c:33)
==63442==    by 0x193AB5: rrdcalctemplate_link_matching (rrdcalctemplate.c:47)
==63442==    by 0x1A9AAE: rrdset_create_custom (rrdset.c:800)
==63442==    by 0x2A1952: do_proc_diskstats (proc_diskstats.c:1312)
==63442==    by 0x29D247: proc_main (plugin_proc.c:153)
==63442==
==63442== 148 bytes in 2 blocks are definitely lost in loss record 2,398 of 2,707
==63442==    at 0x4842888: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==63442==    by 0x56E767E: strdup (strdup.c:42)
==63442==    by 0x14A721: strdupz (libnetdata.c:159)
==63442==    by 0x191628: rrdcalc_replace_variables (rrdcalc.c:44)
==63442==    by 0x191C4A: rrdcalc_update_rrdlabels (rrdcalc.c:87)
==63442==    by 0x29F579: add_labels_to_disk (proc_diskstats.c:856)
==63442==    by 0x2A19AB: do_proc_diskstats (proc_diskstats.c:1331)
==63442==    by 0x29D247: proc_main (plugin_proc.c:153)
==63442==    by 0x158431: thread_start (threads.c:185)
==63442==    by 0x56D178C: start_thread (pthread_create.c:442)
==63442==    by 0x57528E3: clone (clone.S:100)
==63442==
==63442== 148 bytes in 2 blocks are definitely lost in loss record 2,399 of 2,707
==63442==    at 0x4842888: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==63442==    by 0x14A7B4: mallocz (libnetdata.c:170)
==63442==    by 0x14C26C: find_and_replace (libnetdata.c:1518)
==63442==    by 0x1916B5: rrdcalc_replace_variables (rrdcalc.c:63)
==63442==    by 0x191C4A: rrdcalc_update_rrdlabels (rrdcalc.c:87)
==63442==    by 0x29F579: add_labels_to_disk (proc_diskstats.c:856)
==63442==    by 0x2A19AB: do_proc_diskstats (proc_diskstats.c:1331)
==63442==    by 0x29D247: proc_main (plugin_proc.c:153)
==63442==    by 0x158431: thread_start (threads.c:185)
==63442==    by 0x56D178C: start_thread (pthread_create.c:442)
==63442==    by 0x57528E3: clone (clone.S:100)
==63442==
==63442== 150 bytes in 2 blocks are definitely lost in loss record 2,400 of 2,707
==63442==    at 0x4842888: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==63442==    by 0x14A7B4: mallocz (libnetdata.c:170)
==63442==    by 0x14C26C: find_and_replace (libnetdata.c:1518)
==63442==    by 0x19175E: rrdcalc_replace_variables (rrdcalc.c:69)
==63442==    by 0x191C4A: rrdcalc_update_rrdlabels (rrdcalc.c:87)
==63442==    by 0x29F579: add_labels_to_disk (proc_diskstats.c:856)
==63442==    by 0x2A1680: do_proc_diskstats (proc_diskstats.c:1268)
==63442==    by 0x29D247: proc_main (plugin_proc.c:153)
==63442==    by 0x158431: thread_start (threads.c:185)
==63442==    by 0x56D178C: start_thread (pthread_create.c:442)
==63442==    by 0x57528E3: clone (clone.S:100)
==63442==
==63442== 170 bytes in 2 blocks are definitely lost in loss record 2,453 of 2,707
==63442==    at 0x4842888: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==63442==    by 0x56E767E: strdup (strdup.c:42)
==63442==    by 0x14A721: strdupz (libnetdata.c:159)
==63442==    by 0x191628: rrdcalc_replace_variables (rrdcalc.c:44)
==63442==    by 0x1919D6: rrdsetcalc_link (rrdcalc.c:143)
==63442==    by 0x192641: rrdcalc_add_to_host (rrdcalc.c:420)
==63442==    by 0x192A30: rrdcalc_create_from_template (rrdcalc.c:541)
==63442==    by 0x1939BB: rrdcalctemplate_check_conditions_and_link (rrdcalctemplate.c:33)
==63442==    by 0x193AB5: rrdcalctemplate_link_matching (rrdcalctemplate.c:47)
==63442==    by 0x1A9AAE: rrdset_create_custom (rrdset.c:800)
==63442==    by 0x2A1627: do_proc_diskstats (proc_diskstats.c:1249)
==63442==    by 0x29D247: proc_main (plugin_proc.c:153)
==63442==
==63442== 170 bytes in 2 blocks are definitely lost in loss record 2,454 of 2,707
==63442==    at 0x4842888: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==63442==    by 0x56E767E: strdup (strdup.c:42)
==63442==    by 0x14A721: strdupz (libnetdata.c:159)
==63442==    by 0x191628: rrdcalc_replace_variables (rrdcalc.c:44)
==63442==    by 0x191C4A: rrdcalc_update_rrdlabels (rrdcalc.c:87)
==63442==    by 0x29F579: add_labels_to_disk (proc_diskstats.c:856)
==63442==    by 0x2A1680: do_proc_diskstats (proc_diskstats.c:1268)
==63442==    by 0x29D247: proc_main (plugin_proc.c:153)
==63442==    by 0x158431: thread_start (threads.c:185)
==63442==    by 0x56D178C: start_thread (pthread_create.c:442)
==63442==    by 0x57528E3: clone (clone.S:100)
==63442==
==63442== 170 bytes in 2 blocks are definitely lost in loss record 2,455 of 2,707
==63442==    at 0x4842888: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==63442==    by 0x14A7B4: mallocz (libnetdata.c:170)
==63442==    by 0x14C26C: find_and_replace (libnetdata.c:1518)
==63442==    by 0x1916B5: rrdcalc_replace_variables (rrdcalc.c:63)
==63442==    by 0x191C4A: rrdcalc_update_rrdlabels (rrdcalc.c:87)
==63442==    by 0x29F579: add_labels_to_disk (proc_diskstats.c:856)
==63442==    by 0x2A1680: do_proc_diskstats (proc_diskstats.c:1268)
==63442==    by 0x29D247: proc_main (plugin_proc.c:153)
==63442==    by 0x158431: thread_start (threads.c:185)
==63442==    by 0x56D178C: start_thread (pthread_create.c:442)
==63442==    by 0x57528E3: clone (clone.S:100)
==63442==
==63442== 201 bytes in 3 blocks are definitely lost in loss record 2,467 of 2,707
==63442==    at 0x4842888: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==63442==    by 0x14A7B4: mallocz (libnetdata.c:170)
==63442==    by 0x14C26C: find_and_replace (libnetdata.c:1518)
==63442==    by 0x1916B5: rrdcalc_replace_variables (rrdcalc.c:63)
==63442==    by 0x191C4A: rrdcalc_update_rrdlabels (rrdcalc.c:87)
==63442==    by 0x1A3321: rrdset_update_rrdlabels (rrdlabels.c:953)
==63442==    by 0x2AE1DE: do_proc_net_dev (proc_net_dev.c:948)
==63442==    by 0x2AF761: netdev_main (proc_net_dev.c:1511)
==63442==    by 0x158431: thread_start (threads.c:185)
==63442==    by 0x56D178C: start_thread (pthread_create.c:442)
==63442==    by 0x57528E3: clone (clone.S:100)
==63442==
==63442== 207 bytes in 3 blocks are definitely lost in loss record 2,469 of 2,707
==63442==    at 0x4842888: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==63442==    by 0x56E767E: strdup (strdup.c:42)
==63442==    by 0x14A721: strdupz (libnetdata.c:159)
==63442==    by 0x191628: rrdcalc_replace_variables (rrdcalc.c:44)
==63442==    by 0x1919D6: rrdsetcalc_link (rrdcalc.c:143)
==63442==    by 0x192641: rrdcalc_add_to_host (rrdcalc.c:420)
==63442==    by 0x192A30: rrdcalc_create_from_template (rrdcalc.c:541)
==63442==    by 0x1939BB: rrdcalctemplate_check_conditions_and_link (rrdcalctemplate.c:33)
==63442==    by 0x193AB5: rrdcalctemplate_link_matching (rrdcalctemplate.c:47)
==63442==    by 0x1A9AAE: rrdset_create_custom (rrdset.c:800)
==63442==    by 0x2AE1C4: do_proc_net_dev (proc_net_dev.c:933)
==63442==    by 0x2AF761: netdev_main (proc_net_dev.c:1511)
==63442==
==63442== 207 bytes in 3 blocks are definitely lost in loss record 2,470 of 2,707
==63442==    at 0x4842888: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==63442==    by 0x56E767E: strdup (strdup.c:42)
==63442==    by 0x14A721: strdupz (libnetdata.c:159)
==63442==    by 0x191628: rrdcalc_replace_variables (rrdcalc.c:44)
==63442==    by 0x191C4A: rrdcalc_update_rrdlabels (rrdcalc.c:87)
==63442==    by 0x1A3321: rrdset_update_rrdlabels (rrdlabels.c:953)
==63442==    by 0x2AE1DE: do_proc_net_dev (proc_net_dev.c:948)
==63442==    by 0x2AF761: netdev_main (proc_net_dev.c:1511)
==63442==    by 0x158431: thread_start (threads.c:185)
==63442==    by 0x56D178C: start_thread (pthread_create.c:442)
==63442==    by 0x57528E3: clone (clone.S:100)
==63442==
==63442== 412 bytes in 4 blocks are definitely lost in loss record 2,545 of 2,707
==63442==    at 0x4842888: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==63442==    by 0x14A7B4: mallocz (libnetdata.c:170)
==63442==    by 0x14C26C: find_and_replace (libnetdata.c:1518)
==63442==    by 0x1916B5: rrdcalc_replace_variables (rrdcalc.c:63)
==63442==    by 0x191C4A: rrdcalc_update_rrdlabels (rrdcalc.c:87)
==63442==    by 0x1A3321: rrdset_update_rrdlabels (rrdlabels.c:953)
==63442==    by 0x2AEB05: do_proc_net_dev (proc_net_dev.c:1196)
==63442==    by 0x2AF761: netdev_main (proc_net_dev.c:1511)
==63442==    by 0x158431: thread_start (threads.c:185)
==63442==    by 0x56D178C: start_thread (pthread_create.c:442)
==63442==    by 0x57528E3: clone (clone.S:100)
==63442==
==63442== 420 bytes in 4 blocks are definitely lost in loss record 2,547 of 2,707
==63442==    at 0x4842888: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==63442==    by 0x56E767E: strdup (strdup.c:42)
==63442==    by 0x14A721: strdupz (libnetdata.c:159)
==63442==    by 0x191628: rrdcalc_replace_variables (rrdcalc.c:44)
==63442==    by 0x1919D6: rrdsetcalc_link (rrdcalc.c:143)
==63442==    by 0x192641: rrdcalc_add_to_host (rrdcalc.c:420)
==63442==    by 0x192A30: rrdcalc_create_from_template (rrdcalc.c:541)
==63442==    by 0x1939BB: rrdcalctemplate_check_conditions_and_link (rrdcalctemplate.c:33)
==63442==    by 0x193AB5: rrdcalctemplate_link_matching (rrdcalctemplate.c:47)
==63442==    by 0x1A9AAE: rrdset_create_custom (rrdset.c:800)
==63442==    by 0x2AEADF: do_proc_net_dev (proc_net_dev.c:1179)
==63442==    by 0x2AF761: netdev_main (proc_net_dev.c:1511)
==63442==
==63442== 420 bytes in 4 blocks are definitely lost in loss record 2,548 of 2,707
==63442==    at 0x4842888: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==63442==    by 0x56E767E: strdup (strdup.c:42)
==63442==    by 0x14A721: strdupz (libnetdata.c:159)
==63442==    by 0x191628: rrdcalc_replace_variables (rrdcalc.c:44)
==63442==    by 0x191C4A: rrdcalc_update_rrdlabels (rrdcalc.c:87)
==63442==    by 0x1A3321: rrdset_update_rrdlabels (rrdlabels.c:953)
==63442==    by 0x2AEB05: do_proc_net_dev (proc_net_dev.c:1196)
==63442==    by 0x2AF761: netdev_main (proc_net_dev.c:1511)
==63442==    by 0x158431: thread_start (threads.c:185)
==63442==    by 0x56D178C: start_thread (pthread_create.c:442)
==63442==    by 0x57528E3: clone (clone.S:100)
==63442==
==63442== LEAK SUMMARY:
==63442==    definitely lost: 2,971 bytes in 35 blocks

[MrZammler]

Just a note (thanks to all of you guys) we can leave this out of the release.

[ilyam8]

@MrZammler is there anything you want to add or we can merge the PR?

[MrZammler]

Just wanted to not mess with any patch releases, but since we can cherry pick for them, will merge on Monday.

[ilyam8]

Monday, finally 😄