Setting INFLUXDB_META_DIR causes initialization to run every time the container restarts

[bayesio]

influxdb.zip

I've attached a docker compose file that both shows a system with and without an issue. To reproduce the issue:

1. Run the compose file in ./issue (docker-compose up -d)
2. View the logs to see everything is working fine (docker logs influxdb)
3. Stop the container (docker stop influxdb)
4. Start the container (docker start influxdb)
5. View the logs to see that initialization fails because it is trying to recreate the admin user (docker logs influxdb)
6. cleanup the container (docker-compose down)

If we run through the same process for the ./noissue compose file influxdb will start and restart without issue. The only difference in the compose file we aren't setting INFLUXDB_META_DIR

[jsternberg]

Are you maintaining the /mnt directory anywhere? I would guess that the directory is being created again because it doesn't exist across calls.

[bayesio]

I believe so, but I am not extremely familiar with Linux. If it was being created again date -r /mnt/influxdb/db/meta/meta.db would return a different time correct? I ran the following commands. at point 1 was when I stopped the container and point 2 was after I tried to start the container again and the init script failed.

[bayesio]

So, in an attempt to determine if it was a volume mount I modified my test slightly. I ran docker-compose --entrypoint '/bin/date' run influxdb -r /mnt/influxdb/db/meta/meta.db instead of running commands directly on my docker machine. Still no luck though. The output I got is attached below, but in short same date for meta.db between stopping and starting the container while the init runs and fails on starting the container the second time.

output.txt

[ctwilleager]

I was running into the same problem with Kubernetes and I believe I've isolated the problem. I am deploying InfluxDB using PVCs for the data, write-ahead log, and meta directories different than stock so that all the data in these locations is put onto PVCs. I specify environment variables in my deployment as follows:

• INFLUXDB_DATA_DIR=/influxdb-data/data
• INFLUXDB_DATA_WAL_DIR=/influxdb-data/wal
• INFLUXDB_META_DIR=/influxdb-data/meta

I was hitting this same issue whenever the container was restarted, where initialization would always fail. I believe I've tracked it down to an issue in the init-influxdb.sh script. https://raw.githubusercontent.com/influxdata/influxdata-docker/master/influxdb/1.5/init-influxdb.sh

Specifically this is the offending line: if ( [ ! -z "$INIT_USERS" ] || [ ! -z "$INFLUXDB_DB" ] || [ "$(ls -A /docker-entrypoint-initdb.d 2> /dev/null)" ] ) && [ ! "$(ls -d /var/lib/influxdb/meta 2>/dev/null)" ]; then Since I'm not putting meta in /var/lib/influxdb/meta, initialization will always try to run even if it's already been done before. Thus, after restarting the container, I get the same output as @bayesio when my pod restarts.

I corrected the problem for our use case by adding an if/else check to see if the environment variable for INFLUXDB_META_DIR is being provided:

if [ -z $INFLUXDB_META_DIR ]; then META_DIR="/var/lib/influxdb/meta" else META_DIR=$INFLUXDB_META_DIR fi

Then I changed the offending line as follows: if ( [ ! -z "$INIT_USERS" ] || [ ! -z "$INFLUXDB_DB" ] || [ "$(ls -A /docker-entrypoint-initdb.d 2> /dev/null)" ] ) && [ ! "$(ls -d $META_DIR 2>/dev/null)" ]; then

[ctwilleager]

My modified init script is here: https://pastebin.com/YZSkYmtY

[ELymar]

We are running into the same issue. Any plan to fix the issue in the container script?

[GKupfer]

same issue here as well....

[MicahZoltu]

Despite influxd print-config --key-name bolt-path on a running container pointing at /root/.influxdbv2/influxd.bolt, this doesn't actually appear to be a problem. My problem was something else (drive wasn't being mounted properly).

https://github.com/influxdata/influxdata-docker/blob/368e52310cf5e5229e8e40149b2233faea7f623b/influxdb/2.0/entrypoint.sh#L60

~~This appears to be the source of the issue. In the docker container it points at /root/.influxdbv2/influxd.bolt, which is not one of the directories that has a volume associated with it. This means that on shutdown, the boltdb file will be cleared (because it isn't persisted) which will result in re-initialization.~~

~~### Recommended Solution Move influxd.bolt to /etc/influxdb2` so that it gets persisted across runs.~~

~~### Alternative Solution Make it so the boltdb path is configurable via Docker environment variables, so at least the user can move it (right now it appears to be hard coded).~~