shipit-deploy with a remote workspace

[mrozbarry]

I'd really like to be able to do my build process on my server rather than the local machine. My use case is that we're using travis-ci to run deploys, and travis is incredibly slow. My servers have more than enough power to quickly compile my source and copy it to the correct folder.

When I first looked at shipit, this is what I assumed was the behaviour, but upon inspecting my local /tmp/shipit-build directory (where I set my workspace), I noticed it did all the operations locally.

Furthermore, I thought there would be at least someone else in the world that would also want this behaviour, but I have yet to find anything similar to shipit-remote-deploy or shipit-remote-workspace.

Am I just going against the grain wanting this? Is my use-case not meaningful to anyone but my team?

[mrozbarry]

This is what my config looks like:

const execSync = require('child_process').execSync;

module.exports = function shipitConfig(shipit) {
  require('shipit-deploy')(shipit);
  require('shipit-shared')(shipit);
  require('shipit-nvm')(shipit);
  require('shipit-yarn')(shipit);

  const SSH_PORT = process.env.SHIPIT_SSH_PORT || 22;
  const DEV_BRANCH = process.env.SHIPIT_BRANCH || execSync('git rev-parse --abbrev-ref HEAD') || 'develop';

  shipit.initConfig({
    default: {
      workspace: '/tmp/shipit-build',
      dirToCopy: 'dist',
      keepReleases: 5,
      repositoryUrl: 'https://github.com/mrozbarry/private.git',
      shared: {
        overwrite: true,
        files: [
          '.env',
        ],
      },
      yarn: {
        remote: true,
        cmd: 'build',
      },
      nvm: {
        remote: true,
      },
    },
    development: {
      servers: 'deploy@localhost:' + SSH_PORT, // this is running in a docker container/vm.
      deployTo: '/home/deploy/deployTest',
      branch: DEV_BRANCH,
    },
  });

  shipit.on('nvm:alias-default', function nvmAliasDefault() {
    return shipit.remote('npm install -g yarn');
  });
};

[gregberge]

I understand your problem and I think we should provide a way to do it out of the box with Shipit. It is a common use-case. For now the only way is making a custom task workflow to do it manually. I do not know when we will have time to implement this in Shipit. I am focusing on the release of v4 which is a huge release for Shipit.

[ac-obair]

Hi,

I'm not sure if this is the same issue, but it sounds similar.

I chain pipelines, the dev pipeline building assets and successful completion of that allowing me to run a deployment pipleine. Passing artifacts from one pipeline to another. I don't need to build every time I run the shipit command.

My ideal workflow is

      pipleine 1             pipleine 2 
[build-dev / deploy-dev] >[ deploy-production] (passing same assets)

I can easily do this and I've written a nice shipitfile that allows me to do what I want i.e. rolling over loadbalancers and ensuring each deployment is successful before moving onto the next server. The only issue I'm having is that I can't seem to skip the fetch stage on deploy-production. If I could only block the fetch stage or populate my own workspace and just let shipit handle the release and rollback to the server that'd be really good. As we can do with Cap deploy.

I saw on the docs that leaving "repositoryUrl" empty should skip the fetch stage or at least try to deploy without it. Did I just misunderstand this? Is there a way for me to skip this stage? I've tried setting repositoryUrl to empty string or leaving it out entirely but the fetch stage just fails.

[ac-obair]

I should say I get around this by letting it pull, and just copying what I need into a build dir within that workspace and deploying that without any build instructions.

However the whole fetch process does take time, skipping it would take down my deploy time.

Cheers

[gremo]

I was thinking exactly the same. Coming from PHP world with Deployer (a similar deploy tool), i was expecting that the shallow clone would happen on the server, not on my machine.

For me this is a no-go for shiptit: I use windows and the remote copy fails (something about tmp directory etc.)

By the way, why copy hundred of files to to server instead of cloning them in it, directly?

@mrozbarry can you explain your solution? This is my first time with shipit and I can't understand it fully...

[mrozbarry]

@gremo I never ended up posting any solution, so I'm not sure what you mean.

What I ended up doing for now is writing my own script that issues ssh commands to a server, and tested it against a docker image, then on my staging and production servers.

I haven't been on the project I was using this for in a couple of months, and haven't looked at shipit v4.x, so I'd recommend starting there.

That said, you can probably do what you want with shipit.remote(...) and shipit.local(...).

You don't need to use any official plugins with shipit to clone and deploy from remote, my original bug report was just about what I expected shipit to do by default vs what it actually does.

@neoziro should we close this issue? Not sure if it's relevant, and it might be better to have a clearer feature request rather than my own ramblings from last year.

[gregberge]

I think we could support it, it is just a bunch of scripts to write. I prefer to keep it opened.

[gremo]

@mrozbarry sorry, I misunderstood you code in second post.

@neoziro Is that hard to implement that feature? Or at least make the local workspace dir to work with windows? The error is quite simple: a leading slash / in front of the path to %temp% (i.e. /C:\Path\to\temp).

[gregberge]

Making the local workspace working on Windows is a priority for me, being able to build on remote is another feature.

A fix for Windows is welcome!

[gremo]

Any update on this? Unfortunately I'm not so good at javascript, I can look a try to find the issue... but maybe someone have already spotted it :)

Update, the problem is here:

https://github.com/shipitjs/shipit/blob/master/packages/shipit-deploy/src/tasks/deploy/update.js#L65

Changing that line to (note the node path library instead of path2):

const uploadDirPath = _path.resolve(rsyncFrom, shipit.config.dirToCopy || '')

Solves the issue. Why? The resolve method from https://github.com/medikoo/path2 adds a leading / in front of the path, but in in Windows machine, path could be C:\...... ending with a wrong invalid path!

Why using path2 instead of node path?

[gremo]

Ping @neoziro

[gregberge]

@gremo I am sorry I don't have time to work on it actually. Any PR is welcome.