x-develop/watch - Watching changes in additional_contexts

[tr00st]

Description

Somewhat related to comments made re: watch.source paths being relative to the build.context (see https://github.com/compose-spec/compose-spec/pull/253 )

Assuming it's intended long-term that build context is the basis for a x-develop/watch path, it would be useful to specify a reference to build.additional_contexts to allow syncing of code outside the main build context. Something like a from_context entry (named based on COPY --from= syntax from the Dockerfile, though "watch from" probably makes less sense semantically). Referencing paths outside build_context would also work for this use case, but I'm assuming there's some implementation detail that'd make that a bit more difficult.

For our specific use case, with have a lot of smaller repositories, which are consistently checked out - though the same could be applied to a monorepo as per the example I've dropped in below. Each app/product has a repo, and modules (referenced as a dependency in package.json to keep things separate). In an ideal world, any change made in a module should propagate an update to all dependent containers. We've used bind-mounts to achieve this with limited success, though performance is pretty poor (likely due to most of us running Windows).

I've thrown together a dumbed-down repo that demonstrates the sort of format I'm talking about, because code probably speaks louder than words. In reality, we use yarn link to reference the modules from our app repos, but the same principles should apply here.

https://github.com/tr00st/docker-compose-watch-additional-contexts

• edit - Adding docker version/context/info as requested on the issues intro:

Client:
 Cloud integration: v1.0.33
 Version:           24.0.2
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        cb74dfc
 Built:             Thu May 25 21:53:15 2023
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.20.0 (109717)
 Engine:
  Version:          24.0.2
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       659604f
  Built:            Thu May 25 21:52:17 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
PS C:\Work\Code\docker-compose-watch-additional-contexts> docker context show
default
PS C:\Work\Code\docker-compose-watch-additional-contexts> docker info
Client:
 Version:    24.0.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.5
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.19
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scan.exe
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  v0.12.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 18
  Running: 16
  Paused: 0
  Stopped: 2
 Images: 52
 Server Version: 24.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 11.7GiB
 Name: docker-desktop
 ID: e7dc7e6d-7889-45da-8028-8143d78b6bcf
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

[milas]

@tr00st Thanks for the feedback! You're right that the original spec proposal and implementation are different here. We're working on reconciling things as we improve the preview functionality in Compose, so this is great.

As of now, have you been happy with the paths being relative to the Compose project directory?

[tr00st]

@milas

Not really - as per the original report, the limit to being able to only watch files inside the Compose project directory has been pretty limited when working with modular+polyrepo code. What we'd hoped for was a setup where, given a three-repo setup:

• Module 1
• App 1, uses Module 1
• App 2, also uses Module 1

Making changes in Module 1 would automatically update both App 1 and App 2, allowing the developer see the changes in either (or both) context.

Sadly, we've discontinued using Docker for most of our dev environments here since the original report, given some long-standing WSL bugs (plus poor performance on the non-WSL version under windows) making it untenable, but happy to feedback on hypotheticals for this one (I'm assuming I won't always be working on a Windows machine :) )