Binding local folder to worker in cluster

[jochendemuth]

Using docker-compose.yml to deploy citus cluster. Works great!

Now, in preparation for a larger scale deployment I would like to mount the data directories of the workers to local folders. This would allow me to direct worker data volumes easily to different physical drives/storage.

I started by adding a line into the docker-compose.yml file extending the "volumes:" specification so that it reads: ... volumes: - healthcheck-volume:/healthcheck - /mnt/local:/var/lib/postgresql/data manager: ... For a cluster with a single worker node this obviously works like a charm. However, when starting a cluster with two nodes, the second node mounts the identical local folder and both worker nodes read and write onto the same files - disaster happens!

I was hoping to specify a local subfolder named like the worker. Really, just any way to provide distinct folders to every worker. Reading through the docker-compose.yml specification I could not find a way to do so. E.g. I tried various ways to reference a container name or retrieve a runtime enumeration of worker nodes. But to no avail. E.g. ... volumes: - healthcheck-volume:/healthcheck - /mnt/local/{services.worker.container_name}:/var/lib/postgresql/data manager: ...

I am relatively new to yml in general and docker-compose.yml in particular. Would you know a way to accomplish my goal?

[onurctirtir]

@gurkanindibay, do you have some time to have a look into that ?

[gurkanindibay]

@onurctirtir Thanks for the heads up. However,my plate is full for now. I have urgent issues to fix

[rcronin]

@gurkanindibay - Any update on this? Would be a nice feature to be able to do this

[gurkanindibay]

@rcronin Thanks for your interest still I can not prioritize this issue.