Problem using external mariadb in stack mode

[Foxi352]

Hi. Great project, love it. Thanks for this. Now i switched my docker to swarm mode and wanted to create a monitoring stack with multiple services. On my old setup i started all containers manually with docker run. Now i use a compose v3 file for my stack.

Since migrating to the compose / stack mode i can't get it to connect to my external mariadb service. I correctly set the environment variables, and all services of the stack are in the same network. If i execute a shell in my icinga2 container i can ping mariadb and i can see the MYSQL_ROOT_PASSWORD=<xxxxxx> and DEFAULT_MYSQL_HOST=mariadb environment variables. But icinga always starts and installs it's own mysql daemon.

Is this a bug or am i on a wrong track ?

version: '3.2'
services:

    icinga2:
        hostname: icinga
        image: jordan/icinga2:2.8.0
        
        depends_on:
            - influxdb
            - mariadb
        
        ports:
            - '8083:80'
            - '5665:5665'

        volumes:
            - icinga2:/etc/icinga2
            - icinga2-sessions:/var/lib/php/sessions
            - icinga2-data:/var/lib/icinga2
            - icinga2-web:/etc/icingaweb2
            - icinga2-ssmtp:/etc/ssmtp
            - /etc/localtime:/etc/localtime:ro
        
        environment:
            - DEFAULT_MYSQL_HOST=mariadb
            - MYSQL_ROOT_PASSWORD=<xxxxxx>
            
        deploy:
            mode: global
            restart_policy:
                condition: always
            resources:
                limits:
                    cpus: '0.50'
                    memory: 1gb

    grafana:
        [not relevant here]

    influxdb:
        [not relevant here]
    
    mariadb:
        hostname: mariadb
        image: mariadb:10
        command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
        
        volumes:
            - mariadb:/var/lib/mysql
            - /etc/localtime:/etc/localtime:ro

        environment:
            - MYSQL_ROOT_PASSWORD=<xxxxx>

        deploy:
            restart_policy:
                condition: always
            mode: global
            resources:
                limits:
                    cpus: '0.50'
                    memory: 1gb

[jjethwa]

Hi @Foxi352

What error do you see in the icinga2 or icingaweb2 logs? The local mariadb instance might start up, but should not be used since you've set the DEFAULT_MYSQL_HOST environment variable.

[Foxi352]

I see no error message. The local instance starts up and is used:

Starting MariaDB database server: mysqld ..
[....]

MySQL user 'root' has no password but only allows local connections
MySQL user 'icinga' password set to <xxxxxx>
MySQL user 'icinga2-ido-mysq' password set to <xxxxxx>
MySQL user 'icingaweb2' password set to <xxxxxx>
MySQL user 'director' password set to <xxxxxx>
[...]
2018-05-11 17:49:04,536 INFO success: mysql entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

Inside icinga2 container:

MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| director           |
| icinga             |
| icinga2idomysql    |
| icingaweb2         |
| information_schema |
| mysql              |
| performance_schema |
+--------------------+

Here are the env. variables inside icinga2 container:

root@icinga:/# env
ICINGA2_FEATURE_DIRECTOR_USER=icinga2-director
ICINGA2_FEATURE_GRAPHITE=false
HOSTNAME=icinga
ICINGA2_USER_FULLNAME='HIS monitoring'
ICINGA2_FEATURE_DIRECTOR=true
MYSQL_ROOT_PASSWORD=<xxxxx>
ICINGA2_FEATURE_GRAPHITE_HOST=graphite
PWD=/
HOME=/root
ICINGA2_FEATURE_GRAPHITE_PORT=2003
APACHE2_HTTP=REDIRECT
DEFAULT_MYSQL_HOST=mariadb
TERM=xterm
ICINGA2_FEATURE_DIRECTOR_KICKSTART=true
SHLVL=1
ICINGA2_FEATURE_GRAPHITE_URL=http://graphite
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env

Ping from icinga2 to mariadb:

root@icinga:/# ping mariadb
PING mariadb (10.0.0.4) 56(84) bytes of data.
64 bytes from 10.0.0.4 (10.0.0.4): icmp_seq=1 ttl=64 time=0.048 ms
64 bytes from 10.0.0.4 (10.0.0.4): icmp_seq=2 ttl=64 time=0.064 ms

Connection to mariadb works, here i am listing databases inside mariadb container from icinga2 container:

root@icinga:/# mysql -h mariadb -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 9
Server version: 10.2.14-MariaDB-10.2.14+maria~jessie mariadb.org binary distribution

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
+--------------------+
3 rows in set (0.00 sec)

As you can see there are no icinga databases inside mariadb container.