icinga2-influxdb-grafana
icinga2-influxdb-grafana copied to clipboard
Setup guide for Icinga2 with Grafana integration employing InfluxDB.
Setup guide for Icinga2 with Grafana integration employing InfluxDB
This guide uses Debian 10 "Buster" as base
cat /etc/apt/sources.list
deb http://deb.debian.org/debian buster main
deb-src http://deb.debian.org/debian buster main
deb http://security.debian.org/ buster/updates main
deb-src http://security.debian.org/ buster/updates main
deb http://deb.debian.org/debian buster-updates main
deb-src http://deb.debian.org/debian buster-updates main
deb http://deb.debian.org/debian buster-backports main
deb-src http://deb.debian.org/debian buster-backports main
Setup Icinga2
Add source
wget -O - https://packages.icinga.com/icinga.key | apt-key add -
echo "deb http://packages.icinga.com/debian icinga-buster main" > /etc/apt/sources.list.d/icinga.list
echo "deb-src http://packages.icinga.com/debian icinga-buster main" >> /etc/apt/sources.list.d/icinga.list
Get packages
apt-get update
apt-get install icinga2 monitoring-plugins
Check service status
systemctl status icinga2.service
● icinga2.service - Icinga host/service/network monitoring system
Loaded: loaded (/lib/systemd/system/icinga2.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2017-07-28 13:46:00 BST; 16s ago
Setup Icinga2 MySQL support
Get packages
apt-get install icinga2-ido-mysql mariadb-server mariadb-client
Questions during installation:
- Enable Icinga 2's ido-mysql feature?
No
- Configure database for icinga2-ido-mysql with dbconfig-common?
No
Secure MySQL installation
mysql_secure_installation
Create database and user
mysql -u root -p
CREATE DATABASE icinga2 CHARACTER SET utf8 COLLATE utf8_general_ci;
GRANT ALL ON icinga2.* TO 'icinga2'@'localhost' IDENTIFIED BY 'your-icinga2-pwd';
Import database schema
mysql -p icinga2 < /usr/share/icinga2-ido-mysql/schema/mysql.sql
Enable mysql module
icinga2 feature enable ido-mysql
Configure database access for icinga instance
vim /etc/icinga2/features-available/ido-mysql.conf
- password = "",
- host = "localhost",
+ password = "your-icinga2-pwd",
+ host = "127.0.0.1",
Setup Icinga2 web interface
Get packages
apt-get install apache2 icingacli icingaweb2 icingaweb2-module-monitoring \
php php-intl php-imagick php-gd php-mysql php-curl php-mbstring
Set PHP default timezone
vim /etc/php/7.3/apache2/php.ini
-;date.timezone =
+date.timezone = Europe/Paris
systemctl restart apache2.service
Create database and user
mysql -u root -p
CREATE DATABASE icingaweb2 CHARACTER SET utf8 COLLATE utf8_general_ci;
GRANT ALL ON icingaweb2.* TO 'icingaweb2'@'localhost' IDENTIFIED BY 'your-icingaweb2-pwd';
Import database schema
mysql -p icingaweb2 < /usr/share/icingaweb2/etc/schema/mysql.schema.sql
Enable external command pipe
icinga2 feature enable command
systemctl restart icinga2.service
Create setup token
icingacli setup config directory --group icingaweb2
icingacli setup token create
Setup using web interface
http://your-public-host.name/icingaweb2/setup
Setup Token: Your Token
Next
Modules enabled: Monitoring
Next
Authentication Type: Database
Next
Resource Name: icingaweb_db
Database Type: MySQL
Host: 127.0.0.1
Port: 3306
Database Name: icingaweb2
Username: icingaweb2
Password: your-icingaweb2-pwd
Character Set: utf8
Use SSL: False
Next
Backend Name: icingaweb2
Next
Username: admin
Password: your-admin-pwd
Repeat password: your-admin-pwd
Next
Show Stacktraces: Yes
Show Application State Messages: Yes
User Preference Storage Type: Database
Logging Type: Syslog
Logging Level: Error
Application Prefix: icingaweb2
Facility: user
Next
Backend Name: icinga
Backend Type: IDO
Next
Resource Name: icinga_ido
Database Type: MySQL
Host: 127.0.0.1
Port: 3306
Database Name: icinga2
Username: icinga2
Password: your-icinga2-pwd
Character Set: utf8
Use SSL: False
Next
Transport Name: icinga2
Transport Type: Local Command File
Command File: /var/run/icinga2/cmd/icinga2.cmd
Next
Protected Custom Variables: *pw*,*pass*,community
Next
Setup graphing module Grafana
Add source
wget -O - https://packages.grafana.com/gpg.key | apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" > /etc/apt/sources.list.d/grafana.list
Get packages
apt-get update
apt-get install grafana influxdb influxdb-client
Enable and start Grafana service
systemctl daemon-reload
systemctl enable grafana-server.service
systemctl start grafana-server.service
Create database and user
influx
CREATE DATABASE icinga2;
CREATE USER icinga2 WITH PASSWORD 'your-icinga2-pwd';
Send performance data to influx
icinga2 feature enable influxdb
vim /etc/icinga2/features-enabled/influxdb.conf
- //host = "127.0.0.1"
- //port = 8086
- //database = "icinga2"
- //flush_threshold = 1024
- //flush_interval = 10s
- //host_template = {
- // measurement = "$host.check_command$"
- // tags = {
- // hostname = "$host.name$"
- // }
- //}
- //service_template = {
- // measurement = "$service.check_command$"
- // tags = {
- // hostname = "$host.name$"
- // service = "$service.name$"
- // }
- //}
+ host = "127.0.0.1"
+ port = 8086
+ database = "icinga2"
+ username = "icinga2"
+ password = "your-icinga2-pwd"
+ enable_send_thresholds = true
+ enable_send_metadata = true
+ flush_threshold = 1024
+ flush_interval = 10s
+ host_template = {
+ measurement = "$host.check_command$"
+ tags = {
+ hostname = "$host.name$"
+ }
+ }
+ service_template = {
+ measurement = "$service.check_command$"
+ tags = {
+ hostname = "$host.name$"
+ service = "$service.name$"
+ }
+ }
systemctl restart icinga2.service
Navigate to Grafana web interface
Url: http://your-public-host.name:3000
Username: admin
Password: admin
Create new Grafana datasource
Add Datasource: http://your-public-host.name:3000/datasources/new?gettingstarted
Name: InfluxDB
Type: InfluxDB
Default: Yes
Url: http://127.0.0.1:8086
Access: Server (Default)
Database: icinga2
User: icinga2
Password: your-icinga2-pwd
Import Grafana dashboard
Import Dashboard: http://your-public-host.name:3000/dashboard/import
Paste JSON from https://raw.githubusercontent.com/Mikesch-mp/icingaweb2-module-grafana/v1.1.8/dashboards/influxdb/base-metrics.json
Add Icinga Web Grafana module
cd /usr/share/icingaweb2/modules
wget -qO- https://github.com/Mikesch-mp/icingaweb2-module-grafana/archive/v1.3.5.tar.gz | tar xvz
mv icingaweb2-module-grafana-1.3.5 grafana
mkdir /etc/icingaweb2/modules/grafana
Configure Icinga Web Grafana module
Module configuriation (config.ini)
vim /etc/icingaweb2/modules/grafana/config.ini
Graph configuriation (graphs.ini)
vim /etc/icingaweb2/modules/grafana/graphs.ini
Enable anonymous access
vim /etc/grafana/grafana.ini
[auth.anonymous]
# enable anonymous access
-;enabled = false
+enabled = true
# set to true if you want to allow browsers to render Grafana in a <frame>, <iframe>, <embed> or <object>. default is false.
-;allow_embedding = false
+allow_embedding = true
systemctl restart grafana-server.service
Enable Icinga Web Grafana module
icingacli module enable grafana
chown -R www-data:icingaweb2 /etc/icingaweb2
Disable graphing for commands without graphs
Go to the service configuration and set the custom var grafana_graph_disable
for all services, which have no Grafana graph: ssh
, http
, disk
, and icinga
.
vim /etc/icinga2/conf.d/services.conf
+ vars.grafana_graph_disable = true
systemctl restart icinga2.service
Setup SNMP daemon
Get packages
apt-get install snmp snmpd libsnmp-dev
Configure and setup user
systemctl stop snmpd.service
net-snmp-config --create-snmpv3-user -ro -A "your-secret-auth-pwd" -X "your-secret-priv-pwd" -a SHA -x AES snmp
vim /etc/snmp/snmpd.conf
- rocommunity public default -V systemonly
- rocommunity6 public default -V systemonly
+# rocommunity public default -V systemonly
+# rocommunity6 public default -V systemonly
- rouser authOnlyUser
+# rouser authOnlyUser
- trapsink localhost public
+# trapsink localhost public
-iquerySecName internalUser
-rouser internalUser
-defaultMonitors yes
-linkUpDownNotifications yes
+#iquerySecName internalUser
+#rouser internalUser
+#defaultMonitors yes
+#linkUpDownNotifications yes
systemctl start snmpd.service
Add custom check to SNMP
Script to monitor open file handles: check_open_files
apt-get install bc
mkdir -p /usr/local/lib/nagios/plugins
vim /usr/local/lib/nagios/plugins/check_open_files
chmod +x /usr/local/lib/nagios/plugins/check_open_files
vim /etc/snmp/snmpd.conf
- extend test1 /bin/echo Hello, world!
- extend-sh test2 echo Hello, world! ; echo Hi there ; exit 35
-#extend-sh test3 /bin/sh /tmp/shtest
+ extend open_files /usr/local/lib/nagios/plugins/check_open_files
systemctl restart snmpd.service
Test it
snmpwalk -v3 -l authPriv -u snmp -a SHA -A "your-secret-auth-pwd" -x AES -X "your-secret-priv-pwd" 127.0.0.1 .1.3.6.1.4.1.8072.1.3.2
Output:
...
iso.3.6.1.4.1.8072.1.3.2.4.1.2.10.111.112.101.110.95.102.105.108.101.115.1 = STRING: "0.27"
iso.3.6.1.4.1.8072.1.3.2.4.1.2.10.111.112.101.110.95.102.105.108.101.115.2 = STRING: "open=1088 max=401859"
Monitor SNMP hosts
Get packages
apt-get install nagios-snmp-plugins libdigest-hmac-perl
Configure Icinga and Icinga Web Grafana module
Hosts configuriation (hosts.conf)
vim /etc/icinga2/conf.d/hosts.conf
Services configuriation (services.conf)
vim /etc/icinga2/conf.d/services.conf
Groups configuriation (groups.conf)
vim /etc/icinga2/conf.d/groups.conf
Graph configuriation (graphs.ini)
vim /etc/icingaweb2/modules/grafana/graphs.ini
systemctl restart icinga2.service
Get a list of network interfaces
In case you experience issues with the network interface (Unknown interface eth0
), use the following command to get a list of available network interfaces:
/usr/lib/nagios/plugins/check_snmp_int.pl -x your-secret-auth-pwd -p 161 -n eth0 -l snmp -X your-secret-priv-pwd -L sha,aes -H 127.0.0.1 -v
Import Grafana SNMP dashboard
Import Dashboard: http://your-public-host.name:3000/dashboard/import
Use the grafana-snmp-dashboard.