[BUG] `watch` `sync` does not pass project files to container / does not reflect file changes between container restarts

[edvordo]

Description

I've wanted to try the newest docker compose watch (released with latest desktop), but I have two issue with it. The announcement blog post seems to suggest, that adding a config such as this one in an example repository and running docker compose watch should result in a container named app and the folder /var/www/html should have the contents of the current directory.

Thing is, the directory is empty and only new changes to files are copied over.

This lead me to believe, that I would need to add COPY . /var/www/html to Dockerfile so files are copied in on image build, but this creates a second problem, as any changes I made to project files since image build are not reflected to the container between container restarts (say across multiple days of development).

Anyway, here is a repository with an example setup, including steps to reproduce https://github.com/edvordo/docker-compose-watch-test

I'm more than willing to concede I'm reading the docs wrong, but I've tried support channels like the (unofficial?) discord server and we could not figure out what could be wrong.

---

The issue is the same using the official examples provided here: https://github.com/dockersamples/avatars

While yes, the containers will have content after first build (thanks to the COPY instruction), they will not have the changes made to the project files between container restarts:

• while watch is running, change <title> in web/index.html to whatever
• changes are reflected on localhost:5735 and in container
• kill watch, run docker compose down and run docker compose watch again
• <title> in browser / container will NOT have your changes

Steps To Reproduce

No response

Compose Version

compose version
Docker Compose version v2.22.0-desktop.2

docker-compose version
Docker Compose version v2.22.0-desktop.2

Docker Environment

Client:
 Version:    24.0.6
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.5
    Path:     /Users/edvordo/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.22.0-desktop.2
    Path:     /Users/edvordo/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/edvordo/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/edvordo/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.8
    Path:     /Users/edvordo/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/edvordo/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/edvordo/.docker/cli-plugins/docker-scan
  scout: Docker Scout (Docker Inc.)
    Version:  v1.0.7
    Path:     /Users/edvordo/.docker/cli-plugins/docker-scout

Server:
 Containers: 10
  Running: 6
  Paused: 0
  Stopped: 4
 Images: 59
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.4.16-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 3.838GiB
 Name: docker-desktop
 ID: 73a03b6a-4caf-49e5-947b-af0d67cde38e
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[ndeloof]

watch indeed is just a shortcut workflow so you don't have to rebuild your container image, but this won't replace it outside single container lifecycle

[edvordo]

I don't have a problem with watch per-se, more of the action: sync part of the service configuration.

It seems highly impractical to rebuild the container(s) every single time I bring up the containers just so that I get to have changes in since last build. And I can't volume in the directory, because then action: sync / watch will complain, that the directory is already in and won't work.

WARN[0002] path '/path/to/project/docker-compose-watch-test' also declared by a bind mount volume, this path won't be monitored!

I'm looking for a middle ground really, have sync pull / copy the directory into the container on up, like a volume / mount would and have watch still sync in files as I change them.

[SystematicCZ]

I found a sentence from the Docker blog to be quite confusing: 'Docker Compose Watch now automatically builds and starts all required services at launch. One command is all you need: docker compose watch.' (You can read it here: https://www.docker.com/blog/announcing-docker-compose-watch-ga-release/)

However, when I use the watch command, it doesn't rebuild my container if it already exists. So, in reality, I need two commands: compose build and compose watch. This is fine by me, but it's not what I expected after reading the article and the documentation.

[SystematicCZ]

Also wording on the command itself is misleading

Usage:  docker compose watch [SERVICE...]

Watch build context for service and rebuild/refresh containers when files are updated

Options:
      --dry-run   Execute command in dry run mode
      --no-up     Do not build & start services before watching <----- this suggests that containers are rebuild
      --quiet     hide build output

[ndeloof]

The challenge here is that we have no (simple) way to compare local files with those inside container and image used to create one. From a technical standpoint we could check all files in target using the ContainerArchiveInfo and compare with local watch source to detect a mismatch, but this would trigger hundreds API calls.

[edvordo]

Unfortunately, I don't know enough about docker internal workings and I currently do not have the time resources to allocate to learn in order to help here in any way.

I don't technically need to "sync" only changed files, just take everything from host path definition and put it to the target location in container (I realise this is probably a gross oversimplification) and go from there. Is a comparison for every single file required?

[kimdcottrell]

A temporary workaround to this is to open an additional new terminal session and run docker compose cp . app:/var/www/html after watch is running.

I wanted to get a PR out for this, tho I can't quite determine why the calling *composeService.Copy() in Watch() doesn't have the same behavior as the cli command above. The copy occurs, but the container doesn't get updated. I had hoped to do something like: if the watch config action is set to sync or sync+restart and a new watcher is created, run a copy from the watch source path to the watch target.

[thomastweets]

Same problem over here. As is, the docker compose watch feature is a great direction but usable in most of our use cases. I followed reports and blog posts about this new feature closely and we are using docker compose heavily in our development team, however, it not have crossed my mind that this would not take care of syncing the initial state of the configured files and folders at container startup. My guess is that a lot of people will try to use docker compose watch as a direct replacement for bind mounts (e.g. in OSes with inefficiencies resulting from bind mounts) and will run into this very same problem. And due to the different experience that bind mounts give, a lot of users might have a hard time tracking this down.

I will try to go for a coordinated docker compose cp like @kimdcottrell suggested - my initial tests for this looked good. Thanks!

An MR with maybe a flag to decide if one wants to copy current state of the synced folder on container startup would be very nice! 👍

[edvordo]

At least for me, docker compose cp is a no go as it caused file permissions issues, like log files (in a symfony application) could not be written. Due to time constraints I haven't had the opportunity to debug why yet, just adding my two cents that it may not be as straight forward of a solution as it looks.

[glours]

Hey @edvordo @SystematicCZ @thomastweets @kimdcottrell 👋 Can you test this PR #11213 and let me know if this fixes, at least, the issue of the initial sync for you?

[ryuheechul]

I think the problem stems from the container does not sync on the (re)start which I think it should. Why wait for a change to be made to sync only on "newly updated" (requires manual touch if not actually updating them) files after the start? This brings a mismatch between the expectation and what actually happens.

Here is why I think the current behavior is bad

I will explain with a simple scenario:

• image is built with files a, b, c
• after sometime of development, all of those file have new contents
  • let's refer the updated files with a`, b`, c`
  • these are all picked up while watch was running
• however now you do down and watch again and the files are back to a, b, c
  • and you made a change to a` again so now it's a``, b, c in the container
  • which deviated both from the original image and the state of local files
    • it neither reflects the original nor the state of the local files which causes the developers to keep track of the messy state

What I think a solution should be

if watch just pretend (or assume) that all the files for action: sync was updated (mostly likely that's the case with use cases for watch!) on start (and copy them to the right place), the state will meet the expectation and developers shouldn't manually touch **/* to nudge the watch feature.

I think the current state is begging for a solution.

[ndeloof]

The main reason is that doing so, a bunch of files will need to be copied to container on first run/restart. But it doesn't seem we have a better way to address this

[ryuheechul]

(Adding on the above) Can copying unnecessary files (that are not updated) be prevented? e.g. comparing hash (or updated date) between files and only copy the ones are actually updated.

I'm not sure how feasible this is for Docker's case but I'm just sharing what that could be the potential approach to the performance/efficiency concern

[ndeloof]

comparing last updated date require a docker API access for each and every file in the target container. This would have a terrible impact. Pure "replace all" strategy would be more efficient, but still have some impacts

[massimeddu-sj]

In our case, dev containers images don't include the source code, that currently is added via a volume at run-time.

Moving to watch functionality, in our case the "replace all" strategy is preferable as we don't really need to check files inside the image.

If this replace can happen before the ENTRYPOINT is executed, that would be perfect, so the ENTRYPOINT can execute the fresh code at each restart.

[edvordo]

Is there a way to check when an image was last built? If so, maybe do the comparison / sync for the files that are modified later than that date. Maybe rebuild if the file count reaches some large number that it would not be manageable in timely manner.

[ndeloof]

@edvordo oh indeed, I remember we had this discussion and went to the same conclusion 😅 indeed, image build time compared to local file last modified time could trigger a "sync"

[mvhatch]

In our case, dev containers images don't include the source code, that currently is added via a volume at run-time.

Moving to watch functionality, in our case the "replace all" strategy is preferable as we don't really need to check files inside the image.

If this replace can happen before the ENTRYPOINT is executed, that would be perfect, so the ENTRYPOINT can execute the fresh code at each restart.

I vote for this approach as well 🙏

[Bessonov]

Probably, I've encountered one of the issues.

What I want: I have an nginx container and want it to restart when nginx.conf on the host changes.

Expectation 1:

Container should restart if ./tasks/nginx.conf is changed.

services:
  gateway:
    image: nginx:1.25.4-alpine3.18
    volumes:
      - ./tasks/nginx.conf:/etc/nginx/nginx.conf:ro
    develop:
      watch:
        - path: ./tasks/nginx.conf
          action: restart
[...]

Result: Error - action must be one of the following: "rebuild", "sync", "sync+restart". So, "restart" doesn't work.

Expectation 2:

The file is copied on startup, and the container restarts upon changes.

services:
  gateway:
    image: nginx:1.25.4-alpine3.18
    develop:
      watch:
        - path: ./tasks/nginx.conf
          action: sync+restart
          target: /etc/nginx/nginx.conf
[...]

Result: The /etc/nginx/nginx.conf file is from the image, not my host. Changes aren't synced on update, and the container doesn't restart.

Expectation 3:

The file is identical, and the container should restart upon change.

services:
  gateway:
    image: nginx:1.25.4-alpine3.18
    restart: unless-stopped
    volumes:
      - ./tasks/nginx.conf:/etc/nginx/nginx.conf:ro
    develop:
      watch:
        - path: ./tasks/nginx.conf
          action: sync+restart
          target: /etc/nginx/nginx.conf
[...]

Result: The file is indeed identical and appears to be "synced" (probably due to the mount) upon changes. However, the container doesn't restart.

$ docker compose version
Docker Compose version v2.26.1

$ docker version
Client: Docker Engine - Community
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:17:48 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:17:48 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[spidgorny]

Why not running it like this?

docker compose up -d --build && docker compose watch

This should solve the file synchronization issue, right? Your build would be mostly cached, so it will not take a long time to re-build.

Alternatively there's a -watch flag in the docker compose up -d --build --watch, which should re-build and then watch.

[hmans]

Why not running it like this?

Yup, for anyone coming here for a solution, the best workaround at the moment is to just do

docker compose up -w --build

and design the Dockerfile to make use of layer caching where possible (or use a separate Dockerfile that builds on top of the original and adds another ADD or COPY statement.)

Having said that, this is a big oversight in the feature. The develop branch of the compose yaml should absolutely allow us to perform a one-time sync of an entire directory/specific files.

[Bessonov]

Thanks, @spidgorny, I missed the flag. However, why should I explicitly use a watch flag? I mean, it's already in the compose file ¯\_(⊙︿⊙)_/¯

Anyway:

# docker-compose.override.yml
services:

  gateway:
    image: nginx:1.26.0-alpine3.19
    restart: unless-stopped
    develop:
      watch:
        - path: ./tasks/nginx.conf
          action: sync+restart
          target: /etc/nginx/nginx.conf
[...]

$ docker compose watch
[+] Running 11/11
 ✔ Container myproject-gateway-1                  Running                                                                                                                                                       0.0s 
[...]
none of the selected services is configured for watch, consider setting an 'develop' section

$ docker compose watch gateway 
[+] Running 11/0
[...]
 ✔ Container myproject-gateway-1                  Running
0.0s 
[...]
can't watch service "gateway" without a build context

(╯°□°）╯︵ ┻━┻

[ndeloof]

why should I explicitly use a watch flag?

if you don't, compose up will not enable watch mode, which is optional when running compose up. While you have a watch section in your compose file, you maybe don't always want to run in watch mode, so the flag

[Bessonov]

Thanks, @ndeloof ! However, I don't understand and don't agree with the reason behind this decision. I expect that definitions in the compose file are used by default. If there is a volumes section, then I expect that the defined volumes are mounted. Similarly, if there is a build section, I expect that the image is built if it does not already exist (perhaps even rebuilt, but it's OK to not to do that). Isn't this the entire purpose of the file?

Likewise and CONSISTENTLY, I expect that the watch section is respected. If I don't want it, then either I provide another compose file with overrides/resets or specify a --no-watch CLI option, similar to how I use the --no-build, --no-start, --no-recreate, or many other options.

BTW, I use the docker-compose.override.yml file to add the watch section. Really, I don't want to provide the --watch flag in addition.

[par5er]

Apologies for posting on an old thread, but a simpler method of copying all changes from host to container is to simply run:

$ touch *

on the host. This will mark all files in the current directory as "changed" causing docker to copy them into the container (provided they are not in the ignore list). Slightly quicker to type than the docker cp command mentioned above :)

[rtxa]

We need a copy stage, simply as that, who uses it, already should know that if he has a lot of files, it will slow down things on startup, and we don't mind, the pros are better than the cons.

[jhrotko]

We are introducing in next compose version a possible fix for this. For now we are introducing an extension x-initSync to develop.watch.x-initSync for sync and sync+restart actions. This will check for path files which one have been modified after the image creation and it will copy these files to the container when starting compose watch command

[rtxa]

@jhrotko Genius! Although the name x-initSync looks weird to me, why you didn't use initSync?

[ndeloof]

x-* are custom extensions, which does not require we update compose-spec schema. This allows such a fix to be introduced, and once we get good feedback we can introduce an actual attribute (which we will have to support forever)