Problem with volumes sync

[charlie-wasp]

Hello!

I tried to use your solution to fix filesystem speed on the Docker for Mac and ran into the strange issue: synced directory in my app container stays empty. Here is my docker-compose config:

version: "2"
services:
  backend:
    image: idfly/ruby-app
    container_name: backend.proj.local
    working_dir: /app
    entrypoint: ['bundle', 'exec', 'rails', 's', '-p', '80', '-b', '0.0.0.0']
    env_file:
      - ../config.env
      - ./config.env
    links:
      - 'mysql:mysql.proj.local'
      - 'mail:mail.proj.local'
    volumes:
      - /app
      - ../../data:/data

  backend-bg-sync:
    image: cweagans/bg-sync
    volumes:
      - ../../backend:/source
    volumes_from:
      - backend
    environment:
      - SYNC_DESTINATION=/app
      - SYNC_MAX_INOTIFY_WATCHES=40000
      - SYNC_VERBOSE=1
    privileged: true
...
...

So after running docker-compose up I see the error from my backend service Could not locate Gemfile or .bundle/ directory. I commented out entrypoint in my docker-compose file and ran docker-compose run backend ls -la /app. It printed an empty directory.

Here is the output of docker inspect on bg-sync container

"Mounts": [
            {
                "Name": "9c63170dab44816f94a6778e3172c908db2b9381c3496611e482b15f46ead742",
                "Source": "/var/lib/docker/volumes/9c63170dab44816f94a6778e3172c908db2b9381c3496611e482b15f46ead742/_data",
                "Destination": "/app",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Source": "/Users/charlie/CodeProjects/proj/backend",
                "Destination": "/source",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],

Here is the output of docker inspect on backend service container:

"Mounts": [
            {
                "Name": "9c63170dab44816f94a6778e3172c908db2b9381c3496611e482b15f46ead742",
                "Source": "/var/lib/docker/volumes/9c63170dab44816f94a6778e3172c908db2b9381c3496611e482b15f46ead742/_data",
                "Destination": "/app",
                "Driver": "local",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],

As I can see, they both mounts the same directory. I can see my rails app there, and in the bg-sync container everything seems to be ok. Sync between /source and /app in the bg-sync container and my host directory works properly. But /app in my backend container stays empty.

Can you help me to resolve this issue, please?

I am using Docker for Mac 1.12.1, OS X El Capitan 10.11.16

[pinguinjkeke]

Same here

[nlenkowski]

I'm getting the same issue as well.

version: '2'
services:
  db:
    image: mysql:5.7
    volumes:
      - "./db:/var/lib/mysql"
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: wordpress
      MYSQL_DATABASE: wordpress
      MYSQL_USER: wordpress
      MYSQL_PASSWORD: wordpress

  wordpress:
    depends_on:
      - db
    image: wordpress:latest
    volumes:
      - /var/www/html
    links:
      - db
    ports:
      - "80:80"
    restart: always
    environment:
      WORDPRESS_DB_HOST: db:3306
      WORDPRESS_DB_PASSWORD: wordpress

  bg-sync:
    image: cweagans/bg-sync
    volumes:
      - ./html:/source
    volumes_from:
      - wordpress
    environment:
      - SYNC_DESTINATION=/var/www/html
      - SYNC_MAX_INOTIFY_WATCHES=40000
      - SYNC_VERBOSE=1
    privileged: true

When I make changes in /var/www/html in the wordpress container they are reflected in ./html on the host, as expected. However, when I add/remove files on the host they are not reflected in the wordpress container. It seems the sync is only working in one direction. Perhaps my configuration is to blame?

[BenMcH]

Sorry @charlie-wasp and everyone else with getting back to you all with how I set it up. For me, It's just plug and play really. I do have to run it once, rails will start up but not be able to find the files and exit with code 0, at first, but I let my docker-compose run for about a minute or two so that bg-sync can sync files and then shut it down. After that, starting it up works flawlessly.

It looks like they are working on the issues in Docker for mac, however, so hopefully we won't have to work with this container for long. This is a GREAT workaround for now, but it is a bit finicky to set up with rails.

[DanielSchwiperich]

I encountered the problem if the target dir already exists in the web container. This is no problem with mounting host volumes but with data volumes

[SamoB25]

try waiting a bit for it to load... I found an empty dir when I started it, but minute later it was populated.

[zquintana]

Same here. Never seems to populate, I just get:

==> Generating Unison profile

==> Starting continuous sync.
Warning: No archive files were found for these roots, whose canonical names are:
	/source
	/code
This can happen either
because this is the first time you have synchronized these roots, 
or because you have upgraded Unison to a new version with a different
archive format.  

Update detection may take a while on this run if the replicas are 
large.

Unison will assume that the 'last synchronized state' of both replicas
was completely empty.  This means that any files that are different
will be reported as conflicts, and any files that exist only on one
replica will be judged as new and propagated to the other replica.
If the two replicas are identical, then no changes will be reported.

If you see this message repeatedly, it may be because one of your machines
is getting its address from DHCP, which is causing its host name to change
between synchronizations.  See the documentation for the UNISONLOCALHOSTNAME
environment variable for advice on how to correct this.

Donations to the Unison project are gratefully accepted: 
http://www.cis.upenn.edu/~bcpierce/unison

[cweagans]

You'll definitely need to wait for the first sync to complete before you do anything here. I should have some time next week if anyone is interested in jumping on a hangout or something to debug this in real time. Keep an eye on this issue and I'll update as I have time to assist.

[leymannx]

~By default the sync only runs unidirectional.~

~By default SYNC_PREFER is set to only solve conflicts in one direction. Container files are always preferred. So when you change a file on your host (after the initial sync), this now is a conflict. Since this file already exists in the container. And how it's gonna be solved? Right, it'll be rejected since /source inside the container will always be preferred. By default.~

SYNC_SOURCE (default: /source): The path inside the container which will be used as the source of the file sync. Most of the time, you probably shouldn't change the value of this variable. Instead, just bind-mount your files into the container at /source and call it a day.

SYNC_PREFER (default: /source): Control the conflict strategy to apply when there are conflits. By default the contents from the source folder are left unchanged but there is also the "newer" option to pick up the most recent files.

But you can change that easily. You have to add - SYNC_PREFER=newer or SYNC_PREFER: newer respectively. Here comes a config I'm quite happy with from a project I started. Bidirectional sync:

version: '2'

services:

  multiphp:
    image: leymannx/apache-multiphp
    ports:
      - 8856:8856
      - 8871:8871
    depends_on:
      - mariadb
    volumes:
      - /var/www
    working_dir: /var/www

  mariadb:
    image: mariadb
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: root

  filesync_multiphp:
    image: cweagans/bg-sync
    volumes:
      - ./www:/source
    volumes_from:
      - multiphp
    environment:
      SYNC_DESTINATION: /var/www
      SYNC_MAX_INOTIFY_WATCHES: 400000
      SYNC_VERBOSE: 1
      SYNC_PREFER: newer
      SYNC_EXTRA_UNISON_PROFILE_OPTS: |
        ignore = Name .DS_Store
        ignore = Name {.*,*}.sw[pon]
        ignore = Name supervisord.*
        ignore = Name *.sql
        ignore = Name *.tmp
        ignore = Name vendor
        ignore = Name node_modules
        ignore = Path html
    privileged: true

[leymannx]

And yeah, first time sync really may take some time. Wait some minutes. And have a look at the Activity Monitor app on your Mac. Is hyperkit consuming more than 100% CPU? Then Unison is probably syncing right now. Is it back to around 1% CPU? Then have a look and compare the files in the container and on the host because syncing now should be done.

[leymannx]

@cweagans I think it may be less puzzling if SYNC_PREFER is set to newer by default? As I remember in my case it took quite some time to figure this out myself. I'ld prepare a PR if you are OK with it?

[cweagans]

I'm not opposed to defaulting to newer.

[cweagans]

Following up here to offer some clarity around the issue:

1. Sync is not unidirectional. Unison listens for FS events on both roots and replicates them to the other root.
2. prefer is only for conflict resolution. If you just let the container sync and don't do anything else, at the end of the initial sync, both roots should be identical. Changing a file in one of the roots is not a conflict. That's normal, expected operation. If you were to - for instance - change the same file in both the container and the host directory at the same time (or more generally: before one change finished replicating to the other root), then prefer would kick in to decide which version of the conflicted file to use.
3. That said, I'm not opposed to using newer. The most recently updated file is probably the right one to use. I'll take a look at the PR shortly.

[leymannx]

@cweagans Thanks for clarification! I wonder then why it didn't work for me until I've set prefer=newer. But this then maybe leads us to the root of this issue.