Automate deployments

[pjsier]

Building off of the conversation with @orangemug in #123, it looks like Travis and/or CircleCI could be used more to automate deployments. Instead of updating the main GitHub Pages repo, Pages could be turned on in this repo and a build step could be added that deploys the site to the gh-pages branch so that it will still be available at maputnik.github.io/editor

Alternatively, for branch, tag, and PR builds we could use Surge.sh. S3 is great for builds like this, but Surge is completely free and here's an example of someone using it for a similar workflow. I'm happy to take a stab at setting up the build, but have there been any updates on getting maputnik.com back? That could be used as the custom domain for Surge.sh, otherwise maputnik.github.io/editor could still be used as the main location, with individual builds having surge domains

[orangemug]

So I've thought a little about this also.

I'm reluctant to using other services like surge or S3 just because it's another thing that only one (or a few people) have the keys for. Github static site hosting is amazing because everybody we add to the team has access because it's just github.

Also remember we can treat a github repo just like a static file server, so I don't see the advantage using another service.

What I'd personally like is a build of each tag to end up in a github repo. For that to happen I think we just need to create a deploy key and write the build artifacts to a directory with the latest non alpha/beta semver tag symlinked to /latest something like

latest -> v1.0.2
v2.0.0-beta/
v1.0.2/
v1.0.1/
v1.0.0/
v0.3.1/
v0.2.2/

With each directory containing the builds. As hopefully more people use maputnik releasing a bad version will cause a lot more disruption. This will allow our users to rollback to a previous version if we break anything, although hopefully we wont break things (...we'll definitely break stuff). Bonus it's a nice history of the app.

On a related note it'd be really cool if the we could have the following

????.maputnik.com -> maputnik.github.io/????

So new repos auto magically appear. For example

docs.maputnik.com -> maputnik.github.io/docs

...I'm not sure how to do that though :)

Thoughts and comments please!

Note: @lukasmartinelli this relates to https://github.com/maputnik/editor/issues/177#issuecomment-337648678 so would be good to have your thoughts? I guess any of the DNS stuff mentioned would be down to you and you may not have the time.

[pjsier]

I think that makes sense! We could have Travis or CircleCI (I just have more experience with Travis) deploy the build to /editor in the maputnik.github.io repo, as well as to /editor/v1.0.2, so that the next version would just override what's in the top level of /editor while still retaining the older version. The DNS part sounds interesting, but I'm not sure how to do it either

[orangemug]

So I prefer Circle CI, this is mainly because of issue #123. In the long run my preference would be to move over to Circle CI and drop travis. That way we're getting the benefit of PR builds (as outlined in #123) without the need for another service.

Note: @lukasmartinelli hasn't enabled circle CI yet (he's the maputnik admin).

My main goal is to reduce the number of services we use, this should hopefully make it easier for new contributors, by reducing the moving parts.

Thoughts?

[orangemug]

FYI, I've started dev on this, I'll push a branch tomorrow or Friday for review.

[orangemug]

So I'm a little behind on this, but here is the progress so far https://github.com/orangemug/git-deploy is anyone is interested or has any feedback (cc @pjsier)

You can see it working in the following repos

• https://github.com/orangemug/git-deploy-demo - the master repo
• https://github.com/orangemug/git-build-push-demo-builds - the build repo

I'm writing up tests for git-deploy now, then I'll add it to the maputnik repo.

@lukasmartinelli can you enable circleci for me on the maputnik/editor repo and add a deploy key to maputnik/website.

[orangemug]

Right here is how it's going to work. The tools (git-deploy) are built and ready to be integrated into maputnik.

We have a new repository called maputnik/editor-builds which will store all the builds for the maputnik/editor repository.

maputnik/website and maputnik/desktop contain submodules to the maputnik/editor-builds repository.

The content of maputnik/editor-builds repository is generated by git-deploy via CircleCI whenever a semver tag is added. The directory structure will end up looking something like this

builds
├── 0.1.0
├── 0.1.1
├── 0.2.0
├── 0.3.0-beta
└── latest -> ./0.2.0

I have a second util script (which I'll release shortly) that bumps submodules in repositories to the HEAD commit of a branch. This will update maputnik/desktop to the correct version while also causing github pages to rebuild maputnik/website and hence deploy the new version of the code to the world on https://maputnik.com

So all going well cutting a new release will be as easy as running the following from the maputnik/editor repository.

npm version minor
git push origin master --tags

@lukasmartinelli for this to work I need to add deploy keys to GitHub and CircleCI for the following repositories

• maputnik/editor
• maputnik/editor-builds
• maputnik/website

I'm happy to do this of you give me the correct permissions. Else the details can be found in the GitHub and CircleCI sections of this document https://github.com/orangemug/git-deploy/blob/master/docs/quick-start.md

Any feedback welcome.

Note: I know submodules can be awkward to use sometimes, however they are already used in maputnik/desktop so I've decided to use them in maputnik/website also. Later I may move to updating a package.json so something similar but for now this'll work fine.

[orangemug]

@pathmapper so I've been looking at this and I'm not sure I'm a fan of my original solution. So to get the auto deployment working we'd have to create a 'bot' account and create a private access token (PAT). That PAT would have full repo write access to the maputnik/maputnik.github.io repo.

Instead we could do something similar to what you've done with the desktop build in the editor repo. We could have a definition file something along the lines of

// versions.json
{
  "active": "v1.7.0",
  "versions": [
    "v1.8.0",
    "v1.7.0",
    // ...
  ]
}

Where https://maputnik.github.io/editor points to active. All other versions are available via https://maputnik.github.io/releases/{version} as is currently the case.

This has the advantage of being able to build a v1.9.0 release without making it available https://maputnik.github.io/editor so we can 'smoke test' pre-release, which is something I always do at the moment.

Thoughts?

[pathmapper]

Sounds good to me @orangemug.

In this scenario, where would we store the builds? Are these basically the public.zip files from the release assets?

So the deployment workflow would be the following?

1. Create manually a release in the editor repo, e.g. v1.8.0
2. Manually upload a public.zip file with the v1.8.0 build to the release assets of the editor v1.8.0 release (the build could be taken from the editor ci artifact which is currently named editor).
3. Manually update the editor version number in the desktop repo (https://github.com/maputnik/desktop/blob/master/maputnik.go#L19) and create a new desktop release.
4. Manually upload the 3 desktop artificats created by the desktop ci workflow (e.g. https://github.com/maputnik/desktop/runs/734727176) to the editor release assets of the v1.8.0 release
5. Manually update the active version in the versions.json file to v1.8.0.

This would be fine!

Maybe we should change the name of the public.zip file in the release assets to editor.zip so there is no need to rename the artifact. Then we must adapt the desktop makefile to pull the editor.zip. Or just rename the editor artifact to public.zip.

[orangemug]

Apologies for the delay @pathmapper, I thought I'd already replied to this. So I think we can automate some of this still.

I think the workflow could look like this

1. Update package.json with a new tag, create tag of regexp format ^v[0-9]+\.[0-9]+\.[0-9]+$ pointing to that commit.
2. Workflow creates a draft release automatically
3. If we specified a desktop version in the editor somewhere, we could build against that. So for example the editor relies on v1.1.0 of desktop and would build the editor from the current tag.
4. All assets get uploaded to the release via the workflow
5. Manual step update versions.json in maputnik/maputnik.github.io which will trigger a workflow that will download the release assets from maputnik/editor release and add the to the file tree.

So the manual steps will be

1. Creating the version in maputnik/editor, package.json + git tag
2. Once we're happy with (1), updating the versions.json in maputnik/maputnik.github.io

All else will be automated. How does that sound?

Edit: Another manual step, if release is not a beta release, write the release blog post for the release. We'd have probably already written bits already when releasing the beta releases. However the blog post should be more presentable include screenshots etc... Our celebration for all the hard work we've done and to say thanks to everyone who's helped get us there.

[pathmapper]

Sounds good @orangemug, one thing:

If we specified a desktop version in the editor somewhere, we could build against that. So for example the editor relies on v1.1.0 of desktop and would build the editor from the current tag.

Currently the editor version number is hardcoded in desktop https://github.com/maputnik/desktop/blob/c264cd1771c65586142dd05cb3a74b30673a48dd/maputnik.go#L19

It is used e.g. for the --version option:

./maputnik --version
maputnik version Editor: 1.7.0; Desktop: 1.0.5

So maybe we could remove the hardcoded version number and take somehow the current tag instead (would be also good to have the relevant commit there if the workflow is run for an editor PR).

[orangemug]

So maybe we could remove the hardcoded version number and take somehow the current tag instead (would be also good to have the relevant commit there if the workflow is run for an editor PR).

👍

[HarelM]

Every commit to main deploys a new version to github pages. I generally think this is covered now.