Ease CLI usage for preview apps

[hsablonniere]

Let's discuss how the clever-tools can be used in a CI environment to create, update and delete preview apps.

The situation

• John, Janis and Paul work on project-foobar
• project-foobar is a simple node.js webapp hosted on Clever Cloud
• John writes code for feature one
• Janis writes code for feature two
• Paul wants to test and review ongoing work (feature one AND feature two independently).

The workflow

To solve Paul's problem, we need to expand the concept of a unique staging environment to "multiple automatically managed temporary per-branch preview environments". A concept we'll call "preview apps" :wink:.

Here's the kind of workflow this team wants to have:

• John pushes branch feature-one and creates associated PR/MR
• CI server:
  • creates temporary preview app project-foobar-feature-one on Clever Cloud
  • sets domain to https://feature-one.project-foobar.com
  • (sets env vars)
  • (sets small scaling)
  • deploys latest commit from feature-one on app project-foobar-feature-one
  • links to https://feature-one.project-foobar.com in the PR/MR
• Paul can browse, test and review feature one on https://feature-one.project-foobar.com
  • He asks John to fix something
• Janis pushes branch feature-two and creates associated PR/MR
• CI server:
  • creates temporary preview app project-foobar-feature-two on Clever Cloud
  • sets domain to https://feature-two.project-foobar.com
  • (sets env vars)
  • (sets small scaling)
  • deploys latest commit from feature-two on app project-foobar-feature-two
  • links to https://feature-two.project-foobar.com in the PR/MR
• Paul can browse, test and review feature two on https://feature-two.project-foobar.com
  • He validates feature two
  • feature-two branch is merged into master
  • feature-two branch is deleted
  • PR/MR is marked as merged
• CI server:
  • deletes temporary preview app project-foobar-feature-two from Clever Cloud
  • deploys new merged commit from master on production app
• John updates PR/MR by pushing a new commit to feature-one
  • deploys new commit from feature-one on app project-foobar-feature-one
• Paul can browse, test and review fixed version of feature one on https://feature-one.project-foobar.com
  • He validates feature one
  • feature-one branch is merged into master
  • feature-one branch is deleted
  • PR/MR is marked as merged
• CI server:
  • deletes temporary preview app project-foobar-feature-one from Clever Cloud
  • deploys new merged commit from master on production app

The current solution

Everything described in the workflow example can be done with a CI server like GitLab CI and the new version of the clever-tools (with improved error exit status).

Here's the script to execute when a branch is created:

clever create --type node project-foobar-${CI_COMMIT_REF_SLUG}
sleep 10
clever domain add ${CI_COMMIT_REF_SLUG}.project-foobar.com
clever deploy

NOTE: We need sleep 10 to wait for the CC git repo to be ready to use

Here's the script to execute when a branch is updated (and the preview app is already created):

clever link node project-foobar-${CI_COMMIT_REF_SLUG}
clever deploy -f

NOTE: We use a forced push for the clever deploy

Because most CI servers like GitLab CI will execute the same job for created and updated branch, we need to handle both cases in one like this:

CREATE_CC_APP=false
clever link project-foobar-${CI_COMMIT_REF_SLUG} || CREATE_CC_APP=true
if $CREATE_CC_APP; then clever create --type node project-foobar-${CI_COMMIT_REF_SLUG}; sleep 10; fi
clever domain add ${CI_COMMIT_REF_SLUG}.project-foobar.cleverapps.com || true
clever deploy -f

NOTES:

• clever link will exit status with 1 if no app with this name can be linked
• most CI server will stop the script if one of the commands fails so we're using the CREATE_CC_APP variable to trap the result of clever link
• we should put the clever domain add in the if but it was unreadable so I added a || true

We need a better solution!

This solution feels hacky for several reasons:

• (1) We should have something to do clever link OR clever create
  • Do we need a new command?
  • Do we need an option on clever create?
• (2) The clever link is based on the name of the app which is not guaranteed to be unique in a given org
  • How do we handle this?
• (3) We should be able to set a domain if without any errors if it's already set
  • Do we need a clever domain add with a -f force option?
  • Do we need a new clever domain set command?
• (4) We should not have to sleep 10 to be able to push to the git repo
  • Do we need the clever-tools to internally wait for 10 seconds?
  • Do we need the clever-tools to poll the git repo until success before finishing a clever create?
  • Do we need the API to have this waiting logic?
• (5) We should have add some tags so the console can group those environments of the same app visually (in the future)
  • We cannot manage tags in the clever-tools yet.

A nice solution would look like this:

clever create --link-if-name-exists --type node project-foobar-${CI_COMMIT_REF_SLUG}
clever domain add ${CI_COMMIT_REF_SLUG}.project-foobar.cleverapps.com
clever deploy -f

Let's brainstorm here and move the specific discussions of the 5 potential improvements in dedicated issues afterwards.

[hsablonniere]

I forgot the delete step. When a preview app branch is deleted, the CI server should delete it with this:

DELETE_CC_APP=true
clever link clevergitlabci-${CI_COMMIT_REF_SLUG} || DELETE_CC_APP=false
if $DELETE_CC_APP; then clever delete -y -a clevergitlabci-${CI_COMMIT_REF_SLUG}; fi

This handles any cases where the preview app was already deleted.

• (6) We should have something to do clever delete without doing this
  • How do we handle this? (it's related to the unique name problem)

[urcadox]

(1) We should have something to do clever link OR clever create

How about clever link || clever create?

(2) The clever link is based on the name of the app which is not guaranteed to be unique in a given org

I have no idea.

(3) We should be able to set a domain if without any errors if it's already set

The -f / --force flag sounds good to me (it would only ignore the "this domain already exists" error)

(4) We should not have to sleep 10 to be able to push to the git repo

The repository state is known in the database, not sure it is exposed by the API. If it is, the cli could wait poll the API until the repository is created.

(5) We should have add some tags so the console can group those environments of the same app visually (in the future)

Definitely.

(6) We should have something to do clever delete without doing this

Does not solve the problem but the -a flag should accept an app id (or we should have another flag that would) so we don't have to link an app. In this case though, I don't know how we could improve this.

[divarvel]

Issue https://github.com/CleverCloud/clever-tools/issues/83 is all about removing the need for linking apps and being able to use name / ids everywhere. It's been thoroughly discussed and specified, so I think it'd be better to start from there rather than this ad-hoc case.

[urcadox]

Right, I knew we already discussed this somewhere but did not take the time to look through the issues.

[divarvel]

For (2), clever link can also take an application id to remove ambiguity. Humans can use the name feature, but scripts should use the id as much as possible

[divarvel]

For (4) the application polls the repository itself, and retries as long it's not created, so maybe it could just rely on that, instead of waiting? (maybe the repo poll is broken, but fixing it would be better imo)

[hsablonniere]

@divarvel OK, I see what you mean with #83. Thanks for the reminder :wink:. It is clearly linked to this discussion and needs to be addressed at the same time.

About the humans vs. strings, the thing is, in this exact scenario I described, there is no way for a CI server to reuse the ID of the created preview app the next time the branch is updated without storing the app_id <=> branchName correspondence somewhere. When a CI job starts, it knows about the app and the branch that was created, updated or deleted but nothing else.

I think we'll work on #83 in priority while keeping an eye on this preview app situation.

[GuillaumeLog]

Hi there @hsablonniere, as @vvenance told me to do, I'm posting here. Here is a exhaustive description of my use case and issues, quite really similar to yours.

Goal

In the Continous Integration / Deployment workflow I'm aiming for, I try to set :

1. Tests run for each commit on each branch
2. "Review" environments : 1 env. automatically deployed for each branch, updated when there is a commit on this branch.
3. "Pre-prod" environment : 1 env deployed from master branch, updated when there is a merge.

To achieve that on Gitlab, I use :

• gitlab runners that pilot all tests or deployments steps
• a Kubernetes cluster on which runners and tests are runned
• a Clever Cloud account for "Review" and "Preprod" environments.

To be precise, my environment uses multiple components :

• a Django Back-End Django
• a PostGreSQL database
• a Redis
• a Rabbit-MQ
• Celery workers

For now, I'm focusing on make an environment work from tests to pre-prod with only the two first components.

Environnements

"Pre-prod"

Preprod env is simple to handle as there is only one. Apps and addons names can be set in advance and won't change. We can also be sure that they exist or fail if not.

A corresponding Gitlab CI conf may be :

preprod:
  image: node:latest
  script:
    - npm install -g clever-tools
    - clever link -a $CLEVER_PREPROD_APP_DJANGO
    - clever deploy -f

"Review"

Review env are dynamics and thus much more complex. In fact here are explicit an implicit conditions :

• 1 env. per branch
• env. is updated each time a commit is successful on tests
• Tests and deployment piloting are handled by jobs pipelines in gitab-runners : each pipeline is triggered by commits and independent : it does not "know" env. state or previous pipelines "state"

This means that we need that the env. may contain a unique reference to the project + branch. That "project-branch" reference will be the only common point between multiple commits/pipelines on the same branch. And as I have multiple components, each component name will contain this reference.

The expected behavior for each pipeline should be :

• if "project-branch-django" exists :
  • add parameters to deployment (reset db, seeding, etc)
  • deploy
• else :
  • app creation "project-branch-django"
  • add-on creation "project-branch-pg"
  • add parameters to deployment (env. variables, domain, seeding, etc)
  • deploy

A corresponding Gitlab CI conf may be :

review:
  image: node:latest
  stage: review
  environment:
    on_stop: review_stop
  script:
    - npm install -g clever-tools
    - CLEVER_CREATE=false
    - clever link $project-branch-django || CLEVER_CREATE=true
    - if $CLEVER_CREATE; then
        clever create --type python-a $project-branch-django $project-branch-django;
        clever addon create --link $project-branch-django postgresql-addon $project-branch-pg;
      fi
    - clever deploy -f

review_stop:
  image: node:latest
  stage: review
  environment:
    action: stop
  when: manual
  script:
    - npm install -g clever-tools
    - clever link $project-branch-django;
      clever delete -y;
      clever addon delete -y $project-branch-pg
      || true

Issues

Link and app uniqueness

If clever link fails from any other reason than "app does not exists" such as login issues or else, we don't know that and thus creation commands will be executed. But clever create won't fail if app name already exists. So if the unexpected issue happen on clever link , we will create an app for each commit.

I see two issues here :

1. clever link exists with the same error code whichever may be the reason. And I think it would be really dirty to parse the std output to see which error it is, as text may change.
2. clever create allow multiple app creation with the same name. There is no uniqueness on the app name.

Add-on existence

The clever addon create command won't fail if addon already exists, and will create another one.

And there is no command to check if an add-on already exists.

[GuillaumeLog]

Some solutions I see to solve that (that are not exclusive) :

1. Add an --unique option to clever create and clever addon create that will make creation fail if name given already exists
2. Use different exit values on clever link whereas why it is failing
3. Add a clever addon check-like command

[divarvel]

I think 2. is the most satisfying way, as it reflects the API more closely and does not introduce behaviour specific to the CLI (it would be quite arbitrary, as the web UI does not do anything like it)