influxdata-docker
influxdata-docker copied to clipboard
Setting INFLUXDB_META_DIR causes initialization to run every time the container restarts
I've attached a docker compose file that both shows a system with and without an issue. To reproduce the issue:
- Run the compose file in ./issue (
docker-compose up -d
) - View the logs to see everything is working fine (
docker logs influxdb
) - Stop the container (
docker stop influxdb
) - Start the container (
docker start influxdb
) - View the logs to see that initialization fails because it is trying to recreate the admin user (
docker logs influxdb
) - cleanup the container (
docker-compose down
)
If we run through the same process for the ./noissue compose file influxdb will start and restart without issue. The only difference in the compose file we aren't setting INFLUXDB_META_DIR
Are you maintaining the /mnt
directory anywhere? I would guess that the directory is being created again because it doesn't exist across calls.
I believe so, but I am not extremely familiar with Linux. If it was being created again date -r /mnt/influxdb/db/meta/meta.db
would return a different time correct? I ran the following commands. at point 1 was when I stopped the container and point 2 was after I tried to start the container again and the init script failed.
So, in an attempt to determine if it was a volume mount I modified my test slightly. I ran docker-compose --entrypoint '/bin/date' run influxdb -r /mnt/influxdb/db/meta/meta.db
instead of running commands directly on my docker machine. Still no luck though. The output I got is attached below, but in short same date for meta.db between stopping and starting the container while the init runs and fails on starting the container the second time.
I was running into the same problem with Kubernetes and I believe I've isolated the problem. I am deploying InfluxDB using PVCs for the data, write-ahead log, and meta directories different than stock so that all the data in these locations is put onto PVCs. I specify environment variables in my deployment as follows:
- INFLUXDB_DATA_DIR=/influxdb-data/data
- INFLUXDB_DATA_WAL_DIR=/influxdb-data/wal
- INFLUXDB_META_DIR=/influxdb-data/meta
I was hitting this same issue whenever the container was restarted, where initialization would always fail. I believe I've tracked it down to an issue in the init-influxdb.sh script. https://raw.githubusercontent.com/influxdata/influxdata-docker/master/influxdb/1.5/init-influxdb.sh
Specifically this is the offending line:
if ( [ ! -z "$INIT_USERS" ] || [ ! -z "$INFLUXDB_DB" ] || [ "$(ls -A /docker-entrypoint-initdb.d 2> /dev/null)" ] ) && [ ! "$(ls -d /var/lib/influxdb/meta 2>/dev/null)" ]; then
Since I'm not putting meta in /var/lib/influxdb/meta, initialization will always try to run even if it's already been done before. Thus, after restarting the container, I get the same output as @bayesio when my pod restarts.
I corrected the problem for our use case by adding an if/else check to see if the environment variable for INFLUXDB_META_DIR is being provided:
if [ -z $INFLUXDB_META_DIR ]; then META_DIR="/var/lib/influxdb/meta" else META_DIR=$INFLUXDB_META_DIR fi
Then I changed the offending line as follows:
if ( [ ! -z "$INIT_USERS" ] || [ ! -z "$INFLUXDB_DB" ] || [ "$(ls -A /docker-entrypoint-initdb.d 2> /dev/null)" ] ) && [ ! "$(ls -d $META_DIR 2>/dev/null)" ]; then
My modified init script is here: https://pastebin.com/YZSkYmtY
We are running into the same issue. Any plan to fix the issue in the container script?
same issue here as well....
Despite influxd print-config --key-name bolt-path
on a running container pointing at /root/.influxdbv2/influxd.bolt
, this doesn't actually appear to be a problem. My problem was something else (drive wasn't being mounted properly).
https://github.com/influxdata/influxdata-docker/blob/368e52310cf5e5229e8e40149b2233faea7f623b/influxdb/2.0/entrypoint.sh#L60
~~This appears to be the source of the issue. In the docker container it points at /root/.influxdbv2/influxd.bolt
, which is not one of the directories that has a volume associated with it. This means that on shutdown, the boltdb file will be cleared (because it isn't persisted) which will result in re-initialization.~~
~~### Recommended Solution
Move influxd.bolt
to /etc/influxdb2` so that it gets persisted across runs.~~
~~### Alternative Solution Make it so the boltdb path is configurable via Docker environment variables, so at least the user can move it (right now it appears to be hard coded).~~