Freika
Large import lost if container is restarted
Hi,
I started a Google Takeout import using Rake. The Rake job completed, and Sidekiq showed approximately 2.3 million pending tasks. When I took the Docker container down to reboot the host and set it back up again, all pending Sidekiq tasks were gone.
My expectation was that if the pending tasks would be kept in the state and survive container rebuilds/restarts.
My Redis and Postgres volumes are correctly persisted according to the current documentation, which says that only /var/shared/redis should be persisted in the Redis container.
Pending tasks are being stored in Redis, so they should not have been cleared. Although I'll try to reproduce it on my local env to see if it's a real problem and see what I can do about it
Had this same issue with the default docker-compose script. Found that the redis container that was created was referencing a bind map for /var/shared/redis while inside the container itself redis is writing its data to /data. This meant that the redis data was being written to ephemeral storage inside the container so if your redis container restarted before the import was complete, the import would die.
I can confirm that the data is not lost if you add a volume to /data in compose file.
Respectfully, this is not a contributing factor in my case. All other data persists. Only my large import was lost.
Respectfully, this is not a contributing factor in my case. All other data persists. Only my large import was lost.
Yes, this is precisely what my fix does; retains the import data after the Redis container restarts. The import is initially dumped to the Redis container before it's imported into the main database. Realizing that I discovered that the data for the Redis container was being written to nonpersistent storage. My change will create persistent storage where Redis is trying to write and thus when the container restarts the data is not lost.
If your large import is already lost, then you'll have to do it again. I recommend implementing my fix to the docker compose file in order to retain that data.
Confirmed it worked by doing a large import myself that took over a week. The containers restarted several times during that period but continued where they'd left off.
My apologies: I misunderstood @tabacha’s comment to suggest that I had no bind mounts. I now understand that they were referring to the additional Redis mount.
@Freika, I now see that the Redis container documentation explicitly lists the /data volume:
If persistence is enabled, data is stored in the
VOLUME /data, ...
As such, it is perhaps a bug that the docker-compose.yml file included in the repository does not persist this directory across container rebuilds.
Hmm...looking more closely, it doesn’t seem that Dawarich’s docker-compose file actually enables Redis persistence. @SinisterCrayon, have you enabled this manually? Did you add a --save command line option to the Redis container?
Hmm...looking more closely, it doesn’t seem that Dawarich’s
docker-composefile actually enables Redis persistence. @SinisterCrayon, have you enabled this manually? Did you add a--savecommand line option to the Redis container?
I did not. I just did a bind mount modification where instead of /var/shared/redis I changed the bind map to /data.