Add example for local commands and directory upload to documentation

[alexander-schranz]

I see some API command for running task locally on not on the remote but not 100% clear how it can be used or how the commands should be in the deploy.php. Would be great to have an example in the documentation for doing e.g:

npm ci
npm run build

And then upload the build directory e.g.:

local public/build to remote/current/public/build.

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[joelpittet]

I agree, trying to convert 6.x for task()->local()

Example:

task('build:cleanup', function () {
  set('deploy_path', realpath('.') . '/.build');
  $sudo = get('cleanup_use_sudo') ? 'sudo' : '';
  $runOpts = [];
  if ($sudo) {
    $runOpts['tty'] = get('cleanup_tty', FALSE);
  }
  run("$sudo rm -rf {{deploy_path}}", $runOpts);
})->local();

I'm guessing the run() becomes runLocally() and the docs reference calling once() too https://deployer.org/docs/7.x/UPGRADE but I'm not quite sure where/how to translate.

[antonmedv]

It depends on what you are trying to achieve.

[joelpittet]

Local production build of vendor packages to rsync them to remote.

I'll share my 7.x deployer recipes here (computer battery died just before pushing that... so bed time 😉)

[joelpittet]

https://github.com/ubc-cpsc/deployer-recipes/blob/feature/7.x-compatibility/recipes/base.php#L67 where I'm upgrading that snippet above.

And the 6.x version https://github.com/ubc-cpsc/deployer-recipes/blob/main/recipes/base.php#L56

And it's borrowing the "Build Server" strategy from here https://deployer.org/docs/6.x/advanced/deploy-strategies

[antonmedv]

I’m gonna write similar section under ci/cd docs.

[joelpittet]

@antonmedv I think the 7.x shift I think I need to understand is that a Task is run within the Host Context So my deploy task like:

task('deploy', [
  'build',
  'deploy:info',
  'deploy:setup',

Will all try to run in the specified host?

I tried tricking it out with:

task('build', function () {
  // ...
  on(localhost(), function () {
    invoke('deploy:update_code');
    invoke('deploy:vendors');
  });
})->once();

but that yielded:

task build
[stg-www.example.org] /usr/local/bin/php /Users/user/Sites/recipes/vendor/deployer/deployer/bin/dep worker --port 53500 --task build --host stg-www.example.org --decorated -vvv
 InvalidArgumentException  in HostCollection.php on line 20:
  Host "localhost" not found.

@alexander-schranz sorry if I've derailed your original request, LMK if I'm on topic here?

[antonmedv]

I don’t getting, where you took those examples? I never mentioned anywhere what it possible to do it like this.

[joelpittet]

@antonmedv that on() snippet is me just guessing looking at what it does, not from example.

[joelpittet]

I'm just trying to understand the changes in 7.x by example.

[antonmedv]

I’m definitely going to spend some time on documentation.

[lexjwa]

Local() stooped working and also runLocally() throws this error Call to undefined method Deployer\Task\Task::runLocally()

An example will be very helpful.

task('some_task', function () {
   
})->local()->once();

[aaronhuisinga]

We also have a project that makes heavy use of local tasks. We build the project locally, and then upload it to the deploy server.

For example, here is how we build and install our dependencies locally:

desc('Build the project for staging deployment');
task('build', function () use ($buildPath) {
    invoke('build:directory');
    invoke('build:copy');
    within($buildPath, function () {
        writeln('Installing Composer dependencies');
        run('composer install');
        writeln('Installing and compiling NPM dependencies');
        run('yarn');
        run('yarn production');
        run('rm -r node_modules');
    });
})->local();

With the removal of local, I'm not sure at all how to translate something like this to work. Some documentation or pointers for running tasks like this locally would be much appreciated.

[m-graham]

@aaronhuisinga

It looks like the way to achieve this is to do the following:

desc('Build the project for staging deployment');
task('build', function () use ($buildPath) {
    invoke('build:directory');
    invoke('build:copy');
    within($buildPath, function () {
        writeln('Installing Composer dependencies');
        runLocally('composer install');
        writeln('Installing and compiling NPM dependencies');
        runLocally('yarn');
        runLocally('yarn production');
        runLocally('rm -r node_modules');
    });
})->once();

The local function was removed from the task here:

https://github.com/deployphp/deployer/commit/572e487fd5f443552cf08ecf1e2894725f61aef8

One final thing I did notice is that the within command does not appear to be taken into account when running commands locally within it.

[aaronhuisinga]

@m-graham Thanks for the response - I really appreciate the example!

@antonmedv - I would still strongly consider either reverting this change or making it more extensible. We use Deployer for building a few applications locally and then uploaded the built application to the destination server. While the above example would allow us to run commands locally (although with more work required due to the within issue, it seems that it would make interfacing with non-command code difficult or impossible.

For example, here is what our a piece of our current locally run build command looks like in Deployer up until 7.0.0-beta37:

desc('Create build directory if needed');
task('build:directory', function () use ($buildPath) {
    $filesystem = new Filesystem;
    if ($filesystem->isDirectory(getcwd() . '/' . $buildPath)) {
        $filesystem->deleteDirectory(getcwd() . '/' . $buildPath);
    }

    $filesystem->makeDirectory(getcwd() . '/' . $buildPath, 0755, true);
})->local();

desc('Copy application files to a temporary build directory');
task('build:copy', function () use ($buildPath) {
    $filesystem = new Filesystem;
    $files = (new Finder)
        ->in(getcwd())
        ->exclude('.idea')
        ->exclude('.build')
        ->notPath('/^' . preg_quote('tests', '/') . '/')
        ->exclude('node_modules')
        ->ignoreVcs(true)
        ->ignoreDotFiles(false);

    foreach ($files as $file) {
        if ($file->isLink()) {
            continue;
        }

        if ($file->isDir()) {
            $filesystem->makeDirectory($buildPath . '/' . $file->getRelativePathname());
        } else {
            $filesystem->copy($file->getRealPath(), $buildPath . '/' . $file->getRelativePathname());
            $filesystem->chmod($buildPath . '/' . $file->getRelativePathname(), fileperms($file->getRealPath()));
        }
    }
})->local();

desc('Delete build directory after deployment');
task('build:delete', function () use ($buildPath) {
    $filesystem = new Filesystem;
    if ($filesystem->isDirectory(getcwd() . '/' . $buildPath)) {
        $filesystem->deleteDirectory(getcwd() . '/' . $buildPath);
    }
})->local();

We make heavy usage of a few Symfony components for creating a build directory, copying the required files to that directory, and then building and uploading the application. I can't seem to figure out a way to make this work using Deployer versions with the local method removed, which certainly limits what can be done locally and restricts deployment styles like this.

[antonmedv]

@aaronhuisinga I don't see any run() commands in your example. Simply replace local() with once(). It's all that is needed.

Deployer DOES NOT run Symfony components on a remote host. Only commands executed with run().

[aaronhuisinga]

Thanks @antonmedv - you are correct, and it works as expected. It seems the only issue here then is the within method not working correct with the runLocally method as mentioned by @m-graham. I can confirm he is correct and the commands are not run within the specified directory.

[antonmedv]

Yes, this is one thing is changed: within() and cd() affects only run() calls. Thinks of runLocally as advance exec() function.

It's possible to use run() on localhost(), but I think it's overkill.

The reason for removing local() is a lot of other bugs and confusion. (Basically local() was creating temp localhost() for each tasks marked with local(), sets once() and executed. For example: what config will get such task?)

[m-graham]

@antonmedv

One thing that may cause confusion is the logs. I.e. say I have a host named 'remote-host'

task('version', function() { runLocally('php --version'); });

dep version -v

[<remote-host>] run locally php --version
[<remote-host>] PHP 7.4.27 (cli) (built: Dec 14 2021 17:17:06) ( NTS )
[<remote-host>] Copyright (c) The PHP Group
[<remote-host>] Zend Engine v3.4.0, Copyright (c) Zend Technologies
[<remote-host>]     with Zend OPcache v7.4.27, Copyright (c), by Zend Technologies
[<remote-host>]     with Xdebug v3.1.2, Copyright (c) 2002-2021, by Derick Rethans

I think the log should just display [localhost] or the hostname of the localhost if they're being run locally. It doesn't make sense to log the hostname of the remote system for local commands. i.e.

[localhost] run php --version
[localhost] PHP 7.4.27 (cli) (built: Dec 14 2021 17:17:06) ( NTS )
[localhost] Copyright (c) The PHP Group
[localhost] Zend Engine v3.4.0, Copyright (c) Zend Technologies
[localhost]     with Zend OPcache v7.4.27, Copyright (c), by Zend Technologies
[localhost]     with Xdebug v3.1.2, Copyright (c) 2002-2021, by Derick Rethans

[antonmedv]

Actually yes, this makes sense. Will update it.

[joelpittet]

@antonmedv before we were able to do this:

task('build', function () {
  // was other steps but this is the gist...
  set('deploy_path', realpath('.') . '/.build');
  set('deploy_path', realpath('.') . '/.build');
  invoke('deploy:update_code');
  invoke('deploy:vendors');
})->local();

I'm guessing one way to do this is to fork the two tasks and replace all the run with runLocally. The idea behind this is avoid having composer on the servers and adding that outside network traffic and dependency resolution from composer install to the servers from composer.

In trying to forkdeploy:update_code and deploy:vendors with runLocally I end up writing whichLocally for get('bin/git') and commandExistLocally with testLocally. Not working yet but you can see where this is heading...

[antonmedv]

For local builds just invoke composer directly. No need to use tasks deploy:update_code and deploy:vendors.

Main idea: those tasks designed to be used on remote hosts.

It’s in my plan to write a new doc on how to build locally or in CI.

[joelpittet]

@antonmedv ok thanks, I'll give it a try, there are some things I don't need in the tasks when doing locally so performance improvements ;)

[remcotolsma]

One final thing I did notice is that the within command does not appear to be taken into account when running commands locally within it.

Originally posted by @m-graham in https://github.com/deployphp/deployer/issues/2838#issuecomment-1033360699

I also ran into this problem:

within(
	'{{build_path}}',
	function () {
		runLocally( 'composer install --verbose --prefer-dist --no-progress --no-interaction --no-dev --optimize-autoloader' );
	}
);

The composer install command in above example will not run within the {{build_path}} directory.

The runLocally working dir

By default runLocally() commands are executed relative to the recipe file directory. This can be overridden globally by setting an environment variable:

DEPLOYER_ROOT=. dep taskname`

Alternatively the root directory can be overridden per command via the cwd configuration.

runLocally('ls', ['cwd' => '/root/directory']);

https://deployer.org/docs/7.x/cli#the-runlocally-working-dir

So the following code is working:

runLocally(
	'composer install --verbose --prefer-dist --no-progress --no-interaction --no-dev --optimize-autoloader',
	[
		'cwd' => get( 'build_path' ),
	]
);

[ivoba]

I made it work with 2 tasks and 2 hosts, one for local and one for remote.

localhost('local')->set('deploy_path', __DIR__ . '/.build');

host('remote')
    ->setHostname('server-host.com')
    ->set('remote_user', 'user')
    ->set('deploy_path', '/www/blog')
    ->set('http_user', 'user');

....

task('build', function () {
    // build it
})->once();

task('upload', function () {
    upload(__DIR__ . "/.build/current/", '{{deploy_path}}', ['--links']);
});

Then i basically call 2 seperate commands , one for build on local host, one for upload on remote host.

    vendor/bin/dep build local
    vendor/bin/dep upload remote

Not as convenient as previously but it works. :)

[mbrodala]

The snippet mentioned above will work, if you make sure that a localhost() is created and registered first:

localhost(); // Optionally `->set()` anything you need

task('foo', function (): void {
    on(localhost(), function(): void {
        invoke('other:task');
    });
});

This avoids the internal Host "localhost" not found. error thrown by the Deployer master (in HostCollection) which leads to the JSON Error: Syntax error on the Deployer worker.