deploy doesn't handle initializing submodules on the remote

[benlk]

This is only really a problem when deploying submodules for the first time, but the workaround is painfully slow.

$ fab production master verbose deploy
[localhost] Executing task 'production'
[45.79.65.60] Executing task 'master'
On master
[45.79.65.60] Executing task 'verbose'
[45.79.65.60] Executing task 'deploy'
[45.79.65.60] download: <file obj> <- /.git-ftp.log

Warning: get() encountered an exception while downloading '/.git-ftp.log'

Underlying exception:
    No such file

Currently-deployed commit: None
No rollback commit found. Unable to set rollback point.
Checking out branch: master
Deploying...
Mon Nov  7 18:22:11 EST 2016: Host is 'x.x.x.x:2222'.
Mon Nov  7 18:22:11 EST 2016: User is 'redacted'.
Mon Nov  7 18:22:11 EST 2016: Password is set.
Mon Nov  7 18:22:11 EST 2016: Path is './'.
Mon Nov  7 18:22:11 EST 2016: Syncroot is ''.
Mon Nov  7 18:22:11 EST 2016: The remote sha1 is saved in file '.git-ftp.log'.
Mon Nov  7 18:22:11 EST 2016: CACert is ''.
Mon Nov  7 18:22:11 EST 2016: Insecure is ''.
Mon Nov  7 18:22:11 EST 2016: Retrieving last commit from sftp://redacted:***@x.x.x.x:2222/./.

curl: (78) Could not open remote file for reading: No such file or directory
Mon Nov  7 18:22:12 EST 2016: fatal: Could not get last commit. Network down? Wrong URL? Use 'git ftp init' for the initial push., exiting...
Found no existing git repo on ftp host, initializing...
Mon Nov  7 18:22:13 EST 2016: Host is '45.79.65.60:2222'.
Mon Nov  7 18:22:13 EST 2016: User is 'redacted'.
Mon Nov  7 18:22:13 EST 2016: Password is set.
Mon Nov  7 18:22:13 EST 2016: Path is './'.
Mon Nov  7 18:22:13 EST 2016: Syncroot is ''.
Mon Nov  7 18:22:13 EST 2016: The remote sha1 is saved in file '.git-ftp.log'.
Mon Nov  7 18:22:13 EST 2016: CACert is ''.
Mon Nov  7 18:22:13 EST 2016: Insecure is ''.
Mon Nov  7 18:22:13 EST 2016: Check if sftp://redacted:***@x.x.x.x:2222 is accessible.

Mon Nov  7 18:22:17 EST 2016: Check if sftp://redacted:***@x.x.x.x:2222/./ is clean.

curl: (78) Could not open remote file for reading: No such file or directory
Mon Nov  7 18:22:21 EST 2016: Taking all files.
Mon Nov  7 18:22:21 EST 2016: Having files to sync.
Mon Nov  7 18:22:21 EST 2016: There are 8 files to sync:
Mon Nov  7 18:22:21 EST 2016: [1 of 8] Buffered for upload '.git-ftp-ignore'.
Mon Nov  7 18:22:21 EST 2016: [2 of 8] Buffered for upload '.gitignore'.
Mon Nov  7 18:22:21 EST 2016: [3 of 8] Buffered for upload '.gitmodules'.
Mon Nov  7 18:22:21 EST 2016: [4 of 8] Buffered for upload 'README.md'.
Mon Nov  7 18:22:21 EST 2016: [5 of 8] Buffered for upload 'wp-content/plugins/client-hosting-manager'.
Mon Nov  7 18:22:21 EST 2016: Handling submodule sync for wp-content/plugins/client-hosting-manager.
Mon Nov  7 18:22:21 EST 2016: Forced mode enabled.
Mon Nov  7 18:22:21 EST 2016: Host is 'x.x.x.x:2222'.
Mon Nov  7 18:22:21 EST 2016: User is 'redacted'.
Mon Nov  7 18:22:21 EST 2016: Password is set.
Mon Nov  7 18:22:21 EST 2016: Added missing trailing / in path.
Mon Nov  7 18:22:21 EST 2016: Path is './wp-content/plugins/client-hosting-manager/'.
Mon Nov  7 18:22:21 EST 2016: Syncroot is ''.
Mon Nov  7 18:22:21 EST 2016: The remote sha1 is saved in file '.git-ftp.log'.
Mon Nov  7 18:22:21 EST 2016: CACert is ''.
Mon Nov  7 18:22:21 EST 2016: Insecure is ''.
Mon Nov  7 18:22:21 EST 2016: Check if sftp://redacted:***@x.x.x.x:2222 is accessible.

curl: (78) Could not open directory for reading: No such file or directory
Mon Nov  7 18:22:25 EST 2016: fatal: Can't access remote 'sftp://redacted:***@x.x.x.x:2222', exiting...
Mon Nov  7 18:22:25 EST 2016: fatal: Failed to sync submodules.
An error occurred...
Try running fab wp.verify_prerequisites

Done.
Disconnecting from x.x.x.x:2222... done.

The workaround is to manually create the directories on the remote server, because that's what SFTP is trying to catch, but it's not the best.

[benlk]

deploy calls wp.deploy: https://github.com/INN/deploy-tools/blob/master/fablib/init.py#L100

wp.deploy's deploying part is this chunk: https://github.com/INN/deploy-tools/blob/master/fablib/wp/init.py#L173-L184

        if env.get('sftp_deploy', False):
            ret = do_sftp_deploy(env.path)

            if ret.return_code and ret.return_code > 0:
                if ret.return_code in [8, 5, ]:
                    print(colors.cyan("Found no existing git repo on ftp host, initializing..."))
                    ret = initial_deploy(env.path)
                    if ret.return_code and ret.return_code > 0:
                        print(colors.red("An error occurred..."))
                        if not env.verbose:
                            print(colors.yellow(
                                'Try deploying with `verbose` for more information...'))

So what does do_sftp_deploy do, and what are its return codes? What changed that the return codes are no longer in the list [8, 5, ]?

Return codes list is here: https://github.com/git-ftp/git-ftp/blob/master/man/git-ftp.1.md#exit-codes

• 8: not a git project
• 5: error while downloading

I'm thinking that 4 might be relevant as well, error while uploading

do_sftp_deploy essentially just runs the git-ftp command that does the deploy: https://github.com/INN/deploy-tools/blob/master/fablib/wp/init.py#L218-L225

def do_sftp_deploy(dest_path):
    dry_run = '--dry-run ' if env.dry_run else ''
    verbose = '--verbose ' if env.verbose else ''

    command = 'git ftp push %s%s--user "%s" --passwd "%s" sftp://%s:%s/%s' % (
        verbose,
        dry_run,
        env.user,
        env.password,
        env.host_string,
        env.port,
        os.path.normpath(dest_path) + os.sep
    )
    with hide('running', 'warnings'):
        ret = local(command)

    return ret

That looks like: git ftp push --verbose --dry_run --user "$USER" --passwd "$PASSWORD" sftp://$HOST:2222/./

Running that gives back what we see in the normal fab production master verbose deploy logs:

Wed Nov  9 19:34:55 EST 2016: Host is 'host'.
Wed Nov  9 19:34:55 EST 2016: User is 'user'.
Wed Nov  9 19:34:55 EST 2016: Password is set.
Wed Nov  9 19:34:55 EST 2016: Path is './'.
Wed Nov  9 19:34:55 EST 2016: Syncroot is ''.
Wed Nov  9 19:34:55 EST 2016: The remote sha1 is saved in file '.git-ftp.log'.
Wed Nov  9 19:34:55 EST 2016: CACert is ''.
Wed Nov  9 19:34:55 EST 2016: Insecure is ''.
Wed Nov  9 19:34:55 EST 2016: Retrieving last commit from sftp://user:***@host:2222/./.

curl: (67) Authentication failure
Wed Nov  9 19:34:59 EST 2016: fatal: Could not get last commit. Network down? Wrong URL? Use 'git ftp init' for the initial push., exiting...

echo $? returns 5, which is the error code

So it's the correct error code, which means that we're dealing with this chunk: https://github.com/INN/deploy-tools/blob/master/fablib/wp/init.py#L173-L184

                if ret.return_code in [8, 5, ]:
                    print(colors.cyan("Found no existing git repo on ftp host, initializing..."))
                    ret = initial_deploy(env.path)
                    if ret.return_code and ret.return_code > 0:
                        print(colors.red("An error occurred..."))
                        if not env.verbose:
                            print(colors.yellow(
                                'Try deploying with `verbose` for more information...'))

initial_deploy is in the same file, mostly the same code:

def initial_deploy(dest_path):
    dry_run = '--dry-run ' if env.dry_run else ''
    verbose = '--verbose ' if env.verbose else ''

    command = 'git ftp init %s%s--user "%s" --passwd "%s" sftp://%s:%s/%s' % (
        verbose,
        dry_run,
        env.user,
        env.password,
        env.host_string,
        env.port,
        os.path.normpath(dest_path) + os.sep
    )
    with hide('running', 'warnings'):
        ret = local(command)

    return ret

As git ftp init --verbose --dry_run --user "$USER" --passwd "$PASSWORD" sftp://$HOST:2222/./ this returns this with error code 4:

Wed Nov  9 19:36:28 EST 2016: Host is 'host:2222'.
Wed Nov  9 19:36:28 EST 2016: User is 'user'.
Wed Nov  9 19:36:28 EST 2016: Password is set.
Wed Nov  9 19:36:28 EST 2016: Path is './'.
Wed Nov  9 19:36:28 EST 2016: Syncroot is ''.
Wed Nov  9 19:36:28 EST 2016: The remote sha1 is saved in file '.git-ftp.log'.
Wed Nov  9 19:36:28 EST 2016: CACert is ''.
Wed Nov  9 19:36:28 EST 2016: Insecure is ''.
Wed Nov  9 19:36:28 EST 2016: Check if sftp://newsdeskorg-inn-production:***@45.79.65.60:2222 is accessible.

Wed Nov  9 19:36:32 EST 2016: Check if sftp://newsdeskorg-inn-production:***@45.79.65.60:2222/./ is clean.

curl: (78) Could not open remote file for reading: No such file or directory
Wed Nov  9 19:36:34 EST 2016: Taking all files.
Wed Nov  9 19:36:34 EST 2016: Having files to sync.
Wed Nov  9 19:36:34 EST 2016: There are 7 files to sync:
Wed Nov  9 19:36:34 EST 2016: [1 of 7] Buffered for upload '.git-ftp-ignore'.
Wed Nov  9 19:36:34 EST 2016: [2 of 7] Buffered for upload '.gitignore'.
Wed Nov  9 19:36:34 EST 2016: [3 of 7] Buffered for upload '.gitmodules'.
Wed Nov  9 19:36:34 EST 2016: [4 of 7] Buffered for upload 'README.md'.
Wed Nov  9 19:36:34 EST 2016: [5 of 7] Buffered for upload 'wp-content/plugins/client-hosting-manager'.
Wed Nov  9 19:36:34 EST 2016: Handling submodule sync for wp-content/plugins/client-hosting-manager.
Wed Nov  9 19:36:34 EST 2016: Forced mode enabled.
Wed Nov  9 19:36:34 EST 2016: Host is '45.79.65.60:2222'.
Wed Nov  9 19:36:34 EST 2016: User is 'newsdeskorg-inn-production'.
Wed Nov  9 19:36:34 EST 2016: Password is set.
Wed Nov  9 19:36:34 EST 2016: Added missing trailing / in path.
Wed Nov  9 19:36:34 EST 2016: Path is './wp-content/plugins/client-hosting-manager/'.
Wed Nov  9 19:36:34 EST 2016: Syncroot is ''.
Wed Nov  9 19:36:34 EST 2016: The remote sha1 is saved in file '.git-ftp.log'.
Wed Nov  9 19:36:34 EST 2016: CACert is ''.
Wed Nov  9 19:36:34 EST 2016: Insecure is ''.
Wed Nov  9 19:36:34 EST 2016: Check if sftp://newsdeskorg-inn-production:***@45.79.65.60:2222 is accessible.

curl: (78) Could not open directory for reading: No such file or directory
Wed Nov  9 19:36:36 EST 2016: fatal: Can't access remote 'sftp://newsdeskorg-inn-production:***@45.79.65.60:2222', exiting...
Wed Nov  9 19:36:36 EST 2016: fatal: Failed to sync submodules.

So it is that git-ftp runs curl, and curl can't find the remote folder, which it can't find because at the time of the initial deploy the remote folder doesn't exist.

On the other hand, from https://github.com/git-ftp/git-ftp/issues/295 comes this solution:

git submodule foreach git ftp init

With variables in attendance, that would look like git submodule foreach git ftp init --verbose --dry_run --user "$USER" --passwd "$PASSWORD" sftp://$HOST:2222/./, but we'd need to set the correct path on the directory.

$ git submodule foreach pwd
Entering 'tools'
/Users/blk/inn/vms/vagrant-local/www/newsdeskorg/htdocs/tools
Entering 'wp-content/plugins/client-hosting-manager'
/Users/blk/inn/vms/vagrant-local/www/newsdeskorg/htdocs/wp-content/plugins/client-hosting-manager
Entering 'wp-content/themes/largo'
/Users/blk/inn/vms/vagrant-local/www/newsdeskorg/htdocs/wp-content/themes/largo

Something to parse that to get the correct path and then append it to the URL.

[benlk]

Something to parse that to get the correct path

(largo)blk@oyster:~/sites/innsandbox$ git submodule foreach --quiet 'echo $path'
tools
wp-content/plugins/client-hosting-manager
wp-content/themes/largo

This works because submodule foreach has access to certain variables: https://git-scm.com/docs/git-submodule#git-submodule-foreach--recursiveltcommandgt

The command has access to the variables $name, $path, $sha1 and $toplevel: $name is the name of the relevant submodule section in .gitmodules, $path is the name of the submodule directory relative to the superproject, $sha1 is the commit as recorded in the superproject, and $toplevel is the absolute path to the top-level of the superproject.

So the actual command for the foreach init would be

git submodule foreach git ftp init --verbose --dry_run --user "$USER" --passwd "$PASSWORD" sftp://$HOST:2222/./$path

And to determine whether or not that needs to be run, try this:

git submodule foreach 'echo $path' and make sure that ret has results: 316219c

[benlk]

curl: (78) Could not open directory for reading: No such file or directory

git submodule foreach git ftp init doesn't create the directory on the remote. :frowning_face:

dest_path comes from env.path, which is pretty much always ./.

[benlk]

The branch 52-initial-deploy-submodules has a WIP solution for this, but it's incomplete.

The approach is:

1. determine if the repo in question has submodules
2. for each submodule, get the path relative to the git root
3. for each path, use curl to create the folder on the remote server. Using 'cheese' as an example directory, curl sftp://$INNSANDBOX_PRODUCTION_SFTP_HOST:2222/./cheese/ --user $INNSANDBOX_PRODUCTION_SFTP_USER:$INNSANDBOX_PRODUCTION_SFTP_PASSWORD --ftp-create-dirs -Q "MKDIR ./cheese/"
4. then attempt initializing the submodules