deployer Cannot use Deployer with submodules

Before submitting your issue, please make sure that you've checked all of the checkboxes below.

[X] You're running the latest release version of Deployer.
[X] Ensure that you're running at least PHP 7.0.8, you can check this by running php -v
[X] You've ran composer install --no-dev from the root of your installation.
[X] You've ran npm install --production from the root of your installation.
[x] You've ran rm -rf bootstrap/cache/* from the root of your installation.
[X] You have restarted the queue listener and node socket server.

Describe the bug

When trying to use deployer with a repository that has submodules. The deploy fails with the following error: permission denied (publickey) while cloning the submodule.

Expected behaviour

It would properly deploy the project.

Actual behaviour

The deploy fails due to the error mentioned above without event entering the first step.

Steps to reproduce

Include a project with submodule in deployer.
Configure the same ssh key in the master and submodule projects.
Try to deploy a version of the project.

Environment info

Laravel Environment

Laravel Version: 5.5.41
Timezone: America/Sao_Paulo
Debug Mode: ✔
Storage Dir Writable: ✔
Cache Dir Writable: ✔
Decomposer Version: ^1.0
App Size: 2 GB

Server Environment

PHP Version: 7.0.33-0+deb9u3
Server Software: nginx/1.10.3
Server OS: Linux magneto 3.16.0-6-amd64 #1 SMP Debian 3.16.57-2 (2018-07-14) x86_64
Database: pgsql
SSL Installed: ✘
Cache Driver: file
Session Driver: file
Openssl Ext: ✔
PDO Ext: ✔
Mbstring Ext: ✔
Tokenizer Ext: ✔
XML Ext: ✘
Curl Ext: ✘
GD Ext: ✔
JSON Ext: ✔

Installed Packages & their version numbers

andywer/js-localization : dev-laravel-5
backup-manager/laravel : ^1.1
creativeorange/gravatar : ~1.0
doctrine/dbal : 2.5.13
fideloper/proxy : ^3.3
graham-campbell/binput : ~4.0
guzzlehttp/guzzle : ^6.2
htmlmin/htmlmin : ~5.0
intervention/image : ^2.3
laracademy/interactive-make : ^1.1
laravel-notification-channels/hipchat : ~1.0
laravel-notification-channels/twilio : ~2.0.1
laravel-notification-channels/webhook : ~1.1.0
laravel/framework : ^5.5
laravel/tinker : 1.0.*
lubusin/laravel-decomposer : ^1.0
mccool/laravel-auto-presenter : ^6.2
melihovv/laravel-log-viewer : ^5.5
michele-angioni/multi-language : 0.4
pda/pheanstalk : ~3.1
pragmarx/google2fa : ^1.0.0
predis/predis : ~1.0
rebelinblue/laravel5-zxcvbn : ^1.0.0
tymon/jwt-auth : ~0.5
version/version : ^2.2

Extra Information

proc_open enabled : ✔

Logs (see storage/logs/) or other output that would be helpful

[2019-09-05 20:26:38] development.DEBUG: ### Run script locally
bash -s << 'EOF'
    # Turn on quit on non-zero exit
    set -e
    ### Mirror git repository
chmod +x /var/www/deployer/storage/app/tmp/ssh5W8wC8
export GIT_SSH=/var/www/deployer/storage/app/tmp/ssh5W8wC8

if [ ! -d /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git ]; then
    git clone --mirror [email protected]:VE3/colplan.git /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git
fi

cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

git fetch --all --prune
EOF
  
[2019-09-05 20:26:41] development.DEBUG: ### Run script locally
bash -s << 'EOF'
    # Turn on quit on non-zero exit
    set -e
    ### Get commit details - 1365
cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

git log master -n1 --pretty=format:"%H%x09%an%x09%ae"
EOF
  
[2019-09-05 20:26:41] development.DEBUG: ### Run script locally
bash -s << 'EOF'
    # Turn on quit on non-zero exit
    set -e
    ### Create release archive - 1365
git clone --depth 1 --recursive /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git /var/www/deployer/storage/app/tmp/clone_17_20190905202638
cd /var/www/deployer/storage/app/tmp/clone_17_20190905202638
git checkout 36cc87b2cbb01f96473c80abce76c85081a871f0
/var/www/deployer/resources/scripts/tools/GitArchiveAll.sh --tree-ish 36cc87b2cbb01f96473c80abce76c85081a871f0 --format tar.gz --verbose /var/www/deployer/storage/app/17_20190905202636.tar.gz
cd -
rm -rf /var/www/deployer/storage/app/tmp/clone_17_20190905202638
EOF
  
[2019-09-05 20:26:41] development.DEBUG: ### Run script locally
bash -s << 'EOF'
    # Turn on quit on non-zero exit
    set -e
    ### List git references
cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

git tag --list --no-column
EOF
  
[2019-09-05 20:26:43] development.DEBUG: ### Run script locally
bash -s << 'EOF'
    # Turn on quit on non-zero exit
    set -e
    ### List git references
cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

git branch --list --no-column
EOF
  
[2019-09-05 20:26:45] development.ERROR: Cloning into '/var/www/deployer/storage/app/tmp/clone_17_20190905202638'...
warning: --depth is ignored in local clones; use file:// instead.
done.
Submodule 'forecaster' ([email protected]:VE3/forecaster.git) registered for path 'forecaster'
Cloning into '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:VE3/forecaster.git' into submodule path '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster' failed
Failed to clone 'forecaster'. Retry scheduled
Cloning into '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:VE3/forecaster.git' into submodule path '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster' failed
Failed to clone 'forecaster' a second time, aborting

When nailing down the issue I found that cloning a submodule from a mirror requires authentication. So the ssh configs script should be passed as an argument to the CreateReleaseArchive.sh which should export GIT_SSH as you do when you clone the mirror.

If you do not need a submodule just remove --recursive from the git clone command.

Sep 06 '19 20:09 FelipeEmerim

Have you made sure the deploy key is added to the sub module repository?

Stephen Ball [email protected] On 6 Sep 2019, 21:19 +0100, FelipeEmerim [email protected], wrote:

Before submitting your issue, please make sure that you've checked all of the checkboxes below.

• [ X ] You're running the latest release version of Deployer. • [ X ] Ensure that you're running at least PHP 7.0.8, you can check this by running php -v • [ X ] You've ran composer install --no-dev from the root of your installation. • [ X ] You've ran npm install --production from the root of your installation. • [ X ] You've ran rm -rf bootstrap/cache/* from the root of your installation. • [ X ] You have restarted the queue listener and node socket server.

Describe the bug When trying to use deployer with a repository that has submodules. The deploy fails with the following error: permission denied (publickey) while cloning the submodule. Expected behaviour It would properly deploy the project. Actual behaviour The deploy fails due to the error mentioned above without event entering the first step. Steps to reproduce

• Include a project with submodule in deployer. • Configure the same ssh key in the master and submodule projects. • Try to deploy a version of the project.

Environment info Laravel Environment

• Laravel Version: 5.5.41 • Timezone: America/Sao_Paulo • Debug Mode: ✔ • Storage Dir Writable: ✔ • Cache Dir Writable: ✔ • Decomposer Version: ^1.0 • App Size: 2 GB

Server Environment

• PHP Version: 7.0.33-0+deb9u3 • Server Software: nginx/1.10.3 • Server OS: Linux magneto 3.16.0-6-amd64 #1 SMP Debian 3.16.57-2 (2018-07-14) x86_64 • Database: pgsql • SSL Installed: ✘ • Cache Driver: file • Session Driver: file • Openssl Ext: ✔ • PDO Ext: ✔ • Mbstring Ext: ✔ • Tokenizer Ext: ✔ • XML Ext: ✘ • Curl Ext: ✘ • GD Ext: ✔ • JSON Ext: ✔

Installed Packages & their version numbers

• andywer/js-localization : dev-laravel-5 • backup-manager/laravel : ^1.1 • creativeorange/gravatar : ~1.0 • doctrine/dbal : 2.5.13 • fideloper/proxy : ^3.3 • graham-campbell/binput : ~4.0 • guzzlehttp/guzzle : ^6.2 • htmlmin/htmlmin : ~5.0 • intervention/image : ^2.3 • laracademy/interactive-make : ^1.1 • laravel-notification-channels/hipchat : ~1.0 • laravel-notification-channels/twilio : ~2.0.1 • laravel-notification-channels/webhook : ~1.1.0 • laravel/framework : ^5.5 • laravel/tinker : 1.0.* • lubusin/laravel-decomposer : ^1.0 • mccool/laravel-auto-presenter : ^6.2 • melihovv/laravel-log-viewer : ^5.5 • michele-angioni/multi-language : 0.4 • pda/pheanstalk : ~3.1 • pragmarx/google2fa : ^1.0.0 • predis/predis : ~1.0 • rebelinblue/laravel5-zxcvbn : ^1.0.0 • tymon/jwt-auth : ~0.5 • version/version : ^2.2

Extra Information

• proc_open enabled : ✔

Logs (see storage/logs/) or other output that would be helpful [2019-09-05 20:26:38] development.DEBUG: ### Run script locally

bash -s << 'EOF'

Turn on quit on non-zero exit

set -e

Mirror git repository

chmod +x /var/www/deployer/storage/app/tmp/ssh5W8wC8

export GIT_SSH=/var/www/deployer/storage/app/tmp/ssh5W8wC8

if [ ! -d /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git ]; then

git clone --mirror [email protected]:VE3/colplan.git /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

fi

cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

git fetch --all --prune

EOF

[2019-09-05 20:26:41] development.DEBUG: ### Run script locally

bash -s << 'EOF'

Turn on quit on non-zero exit

set -e

Get commit details - 1365

cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

git log master -n1 --pretty=format:"%H%x09%an%x09%ae"

EOF

[2019-09-05 20:26:41] development.DEBUG: ### Run script locally

bash -s << 'EOF'

Turn on quit on non-zero exit

set -e

Create release archive - 1365

git clone --depth 1 --recursive /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git /var/www/deployer/storage/app/tmp/clone_17_20190905202638

cd /var/www/deployer/storage/app/tmp/clone_17_20190905202638

git checkout 36cc87b2cbb01f96473c80abce76c85081a871f0

/var/www/deployer/resources/scripts/tools/GitArchiveAll.sh --tree-ish 36cc87b2cbb01f96473c80abce76c85081a871f0 --format tar.gz --verbose /var/www/deployer/storage/app/17_20190905202636.tar.gz

cd -

rm -rf /var/www/deployer/storage/app/tmp/clone_17_20190905202638

EOF

[2019-09-05 20:26:41] development.DEBUG: ### Run script locally

bash -s << 'EOF'

Turn on quit on non-zero exit

set -e

List git references

cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

git tag --list --no-column

EOF

[2019-09-05 20:26:43] development.DEBUG: ### Run script locally

bash -s << 'EOF'

Turn on quit on non-zero exit

set -e

List git references

cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

git branch --list --no-column

EOF

[2019-09-05 20:26:45] development.ERROR: Cloning into '/var/www/deployer/storage/app/tmp/clone_17_20190905202638'...

warning: --depth is ignored in local clones; use file:// instead.

done.

Submodule 'forecaster' ([email protected]:VE3/forecaster.git) registered for path 'forecaster'

Cloning into '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster'...

Permission denied (publickey).

fatal: Could not read from remote repository.

Please make sure you have the correct access rights

and the repository exists.

fatal: clone of '[email protected]:VE3/forecaster.git' into submodule path '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster' failed

Failed to clone 'forecaster'. Retry scheduled

Cloning into '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster'...

Permission denied (publickey).

fatal: Could not read from remote repository.

Please make sure you have the correct access rights

and the repository exists.

fatal: clone of '[email protected]:VE3/forecaster.git' into submodule path '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster' failed

Failed to clone 'forecaster' a second time, aborting

When nailing down the issue I found that cloning a submodule from a mirror requires authentication. So the ssh configs script should be passed as an argument to the CreateReleaseArchive.sh which should export GIT_SSH as you do when you clone the mirror. If you do not need a submodule just remove --recursive from the git clone command. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

Sep 06 '19 20:09 REBELinBLUE

Yes, same private key on the two repos. I can clone them as long as I pass the ssh key. It seems that the deployer does not use the wrapper ssh script when cloning from the mirror which raises the error.

Sep 06 '19 20:09 FelipeEmerim

A simple test:

#!/usr/bin/env sh

ssh -o CheckHostIP=no \
    -o IdentitiesOnly=yes \
    -o StrictHostKeyChecking=no \
    -o PasswordAuthentication=no \
    -o IdentityFile=MyprivateKey $*`

export GIT_SSH=/path/to/script.sh

git clone --mirror [email protected]:VE3/repoWithSubmodules.git

This is what deployer does to clone the mirror and it is working. Now when it clones from the mirror to the repository to archive, it does not set the git_ssh var so:

unset GIT_SSH

git clone --depth 1 --recursive projectWithSubmdules.git projectWithSubmodules

This last command will raise the mentioned error.

Sep 06 '19 20:09 FelipeEmerim

Hi everybody, I have a same problem. Project with submodules, all of them have access keys same as the main project, but deploy keeps failing.

Is there any way to make it work without getting rid of the submodules?

Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.
fatal: clone of '[email protected]:xyz.git' into submodule path 'xyz' failed

Mar 29 '20 15:03 vhlavsa

I tried some workarounds but none of them worked. In our case, since the submodule is actually a SOAP service we deployed it separately.

Mar 30 '20 19:03 FelipeEmerim

Hey there 👋🏼

I've encountered similar problem some time ago, and ended up writing a quick tip on how I approached it – https://heydarmen.com/blog/using-deployer-with-git-submodules.html

Jun 16 '22 11:06 darmen

hey @darmen I think you are looking for https://github.com/deployphp/deployer

Jun 16 '22 20:06 REBELinBLUE

Sorry @REBELinBLUE.

Jun 17 '22 12:06 darmen

No worries :)

Jun 17 '22 13:06 REBELinBLUE

deployer deployer copied to clipboard

Cannot use Deployer with submodules

Describe the bug

Expected behaviour

Actual behaviour

Steps to reproduce

Environment info

Laravel Environment

Server Environment

Installed Packages & their version numbers

Extra Information

Logs (see storage/logs/) or other output that would be helpful

Turn on quit on non-zero exit

Mirror git repository

Turn on quit on non-zero exit

Get commit details - 1365

Turn on quit on non-zero exit

Create release archive - 1365

Turn on quit on non-zero exit

List git references

Turn on quit on non-zero exit

List git references

deployer
deployer copied to clipboard