Support `ADD` cloning git repository recursive submodules via SSH

[MisterDA]

Description

The Dockerfile ADD instruction can add files from a git repository. My understanding is that it improves caching over RUN git clone.

To add a repository via SSH, whether public or private, you must pass an SSH key for authentication.

ADD [email protected]:user/repo.git /usr/src/things/

If however the git repository submodules can only be fetched via ssh, then, unless I'm mistaken, ADD is unable to fetch them. One has to resort to ssh mounts to be able to do RUN --mount=type=ssh git clone --recursive ....

Could ADD be enhanced to fetch the submodules via SSH, too?

Thanks!

[tonistiigi]

I don't know what exactly is the issue here. ADD allows SSH URLs and submodules should be fetched as well (you may need to enable --keep-git-dir for that.

[MisterDA]

This is essentially my Dockerfile:

# syntax=docker/dockerfile:1
FROM ubuntu:latest
ADD --keep-git-dir ssh://[email protected]:xxxx/repo.git /mnt

which I build with Docker on macOS Sequoia 15.5 (Docker version 28.3.0, build 38b7060)

docker build --ssh default=$SSH_AUTH_SOCK --platform linux/x86_64 - < ./Dockerfile

[stage-0 6/6] ADD --keep-git-dir ssh://[email protected]:xxxx/repo.git /mnt
ref: refs/heads/master HEAD
adc83b19e793491b1c6ea0fd8b46cd9f32e592fc HEAD
adc83b19e793491b1c6ea0fd8b46cd9f32e592fc refs/heads/master
ref: refs/heads/master HEAD
adc83b19e793491b1c6ea0fd8b46cd9f32e592fc HEAD
From ssh://git.example.com:xxxx/repo
* [new branch] master -> master
* [new branch] master -> origin/master
Initialized empty Git repository in /var/lib/desktop-containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/4277/fs/.git/
commit
From file:///var/lib/desktop-containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/4275/fs
* [new tag] master -> master
* [new tag] master -> master
Note: switching to 'FETCH_HEAD'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at adc83b19 Merge branch 'branch-name' into 'master'
Submodule 'submodule' (ssh://[email protected]:xxxx/submodule.git) registered for path 'path_to_submodule'
Cloning into '/var/lib/desktop-containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/4277/fs/path_to_submodule'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'ssh://[email protected]:xxxx/submodule.git' into submodule path '/var/lib/desktop-containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/4277/fs/path_to_submodule' failed
Failed to clone 'path_to_submodule_0'. Retry scheduled
Cloning into '/var/lib/desktop-containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/4277/fs/path_to_submodule'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'ssh://[email protected]:xxxx/submodule.git' into submodule path '/var/lib/desktop-containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/4277/fs/path_to_submodule' failed
Failed to clone 'path_to_submodule' a second time, aborting

To me it seems that the ssh secret key is not picked when cloning the submodules. A RUN --mount=type=ssh git clone --recursive ssh://[email protected]:xxxx/repo.git /mnt works. My ssh config has this:

Host git.example.com
     Port xxxx
     IdentitiesOnly yes
     IdentityFile ~/.ssh/id_ed25519_example

Thanks for any help!

[tonistiigi]

@MisterDA Indeed there seems to be a bug in here. SSH socket is not passed to the submodules request https://github.com/moby/buildkit/blob/v0.23.2/source/git/source.go#L668-L669 like it is passed for the main fetch request https://github.com/moby/buildkit/blob/v0.23.2/source/git/source.go#L752-L753

[MisterDA]

Wow, nice! Is that an easy fix?

[tonistiigi]

I looked into it and these options were indeed passed already.

I tested this with a private repository that has another private repository via SSH URL in .gitmodules and everything seemed to work correctly by just passing --ssh=default to the build.

You obviously can't configure anything with your host SSH config (I can see a snippet in https://github.com/moby/buildkit/issues/6075#issuecomment-3067301169 ) as build sandbox will not read your host config. Everything needs to be configured with the --ssh flag on the build command.

[tonistiigi]

#5 [2/3] ADD [email protected]:tonistiigi/gh-test.git /src
#5 0.056 Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/17/fs/
#5 2.632 ref: refs/heads/main	HEAD
#5 2.782 49690341a90cbe8e61eea06baf9109d90d508d3c	HEAD
#5 5.509 49690341a90cbe8e61eea06baf9109d90d508d3c	refs/heads/main
#5 DONE 5.7s

#5 [2/3] ADD [email protected]:tonistiigi/gh-test.git /src
#5 2.484 ref: refs/heads/main	HEAD
#5 2.653 49690341a90cbe8e61eea06baf9109d90d508d3c	HEAD
#5 5.886 From github.com:tonistiigi/gh-test
#5 5.886  * [new branch]      main       -> main
#5 5.886  * [new branch]      main       -> origin/main
#5 5.933 Submodule 'actions-playground' ([email protected]:tonistiigi/gh-actions-playground2.git) registered for path 'actions-playground'
#5 5.934 Cloning into '/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/19/fs/actions-playground'...
#5 8.810 Submodule path 'actions-playground': checked out '8625637cd3a812565c9355c83e3879c2d8e264ca'
#5 DONE 14.5s

#6 [2/3] ADD [email protected]:tonistiigi/gh-test.git /src
#6 DONE 0.1s

#7 [3/3] RUN ls -la

[MisterDA]

Sorry, I still experience my problem, and I can't figure out how to fix it from your messages. I have only the necessary key loaded into the agent.

[L0rD59]

I tested this with a private repository that has another private repository via SSH URL in .gitmodules and everything seemed to work correctly by just passing --ssh=default to the build.

Hi there,

Seems i have the same issue, not with ADD instruction, but with --build-context argument, and with private https submodule

$ GIT_AUTH_TOKEN=... docker build --progress=plain --ssh default --secret id=GIT_AUTH_TOKEN -t my-app --build-context=python.src=git@git/my-main-repo .

#16 1.171 Cloning into '/var/lib/docker/overlay2/c4z6zu5yr67mfwufmfsafgypd/diff/my_private_submodule'...
#16 1.376 fatal: could not read Username for 'https://github.com': terminal prompts disabled
#16 1.379 fatal: clone of 'https://my_private_submodule.git' into submodule path '/var/lib/docker/overlay2/c4z6zu5yr67mfwufmfsafgypd/diff/my_private_submodule' failed
#16 1.379 Failed to clone 'my_private_submodule'. Retry scheduled

works well for main git context, but not for these submodules

reference : https://docs.docker.com/build/building/secrets/#git-authentication-for-remote-contexts

Thx a lot