Higher level commands

[tailhook]

Motivation

As projects managed by vagga get bigger, we need more tools to manage them easily. Here is a set of ideas in unified discussion to find out how they fit together.

Proposal

1. Using Commands From Different Mixins

When there was only one file we could use anchors to refer to the commands defined elsewhere, but when using mixins we can't. This is mostly useful for supervise commands.

This is #466, but using shorter !Ref syntax:

commands:
  run-a-database: !Command # ...
  other-cmd: !Supervise
    children:
      db: !Ref run-a-database

2. More powerful pipelines

Currently we have:

• prerequisites to run commands sequentially
• !Supervise to run commands in parallel

But what we really want is more than that:

1. Run a database, wait for become available then start the app
2. Run database, run migration suite, start the app
3. Run few test suites in parallel
4. Run many iterations of test suite

And all the combinations of above. This is a tough problem to solve, but let's imagine we can draw a diagram:

run: !Supervise
  pipeline: |
    [ mkdirs ]
              [ ----------- db ------------ ]
              [ ----------- cache --------- ]
              [ wait-db ]             
                         [ migration ]
              [ ----- wait-cache --- ]
                                      [ app ]
  children:
    db: !Command
    cache: !Command
    wait-db: !Command
    wait-cache: !Command
    migration: !Command
    app: !Command

Or maybe using a graph (dot-like syntax, where arrow -> means "depends on", or "after"):

run: !Supervise
  dependencies: |
    {db, cache} -> mkdir
    migration -> wait-db
    app -> {wait-db, wait-cache, migration}
  children:
    db: !Command
    cache: !Command
    wait-db: !Command
    wait-cache: !Command
    migration: !Command
    app: !Command

3. Subprojects

Mixins and subconfigs are great for reusing parts of configs from elsewhere, but they only work when target config is specifically prepared to be sourced. For example, they don't change where /work dir point to. Also subconfigs don't handle commands at all, while mixins handle command conflicts by overriding commands (which is fine for them but not for cross-project embedding).

So this issue proposes subprojects:

subprojects:
  db: ./cookbook/postgres
  images: ./services/pixel

commands:
  run: !Supervise
    prerequisites: [db:migration]
    children:
      db: !Ref db:run
      images: !Ref images:run
      app: !Command # ...

In this case !Ref db:run and db:migration work exactly the same as if you would run vagga run or vagga migration in the cookbook/postgres directory, but allows commands to be used as part of a higher level pipeline.

The subprojects here are just subdirectories. It's expected that they are git submodules in most cases, but no such obligations are done.

One of the questions here is how .vagga folder is managed here? Are these projects glued together, or work exactly as cd'ing into a directory? Separate dirs work with a least surprise for user. While single directory is probably faster and more space-efficient, still having more edge-cases though.

4. Workspaces

The idea is that subprojects should be great, but we also want loose coupling.

Let's imagine we have services B and C depend on A. They both probably have A as a subproject. But then we have a superproject which contains all the A, B, and C. And at a glance it looks like A will be duplicated three times.

This is easy to overcome if you have only one "superproject". Just don't use subprojects for inner project and run anything that involves more than one project in a top level thing. This doesn't work well for integration tests though.

So the idea here is probably to have a parent directory for multiple projects and to remap their subprojects to different folders. I.e. workspace/B/A should be mapped to workspace/A, it's easy to do inside vagga using bind mounts.

While it looks so simple the real questions of how to organize workspace, is it versioned, how to match common subprojects, how to manage git submodules, and so on. So probably we should postpone it until we have more usecases for subprojects to find good solution to the problem.

[alexander-irbis]

where arrow -> means "depends on", or "after"

This is not so obvious. Other people (like me) can read this arrow as "then". Of course we can get used to it in time, but maybe left arrow "<-" would be better if this is allowed by syntax, or maybe to swap the arguments in places?

[tailhook]

Of course we can get used to it in time, but maybe left arrow "<-" would be better if this is allowed by syntax, or maybe to swap the arguments in places?

Sure, this was just a quick sketch. It's currently more of a question of whether we want a graph, a block diagram, or maybe just 'before/after' parameters in commands themselves?

[tailhook]

Another idea about subprojects and workspaces was here: https://github.com/tailhook/vagga/issues/14

[tailhook]

Well, the good part of discussion in #14 is that you might want subprojects be:

1. Specified by git repo
2. Be checked out automatically somewhere in .vagga dir
3. Being possible to override in local settings

So subprojects are specified like this:

subprojects:
  db: git://github.com/our-company/database-repo#v0.1.2
  images: git://gitlab.internal/services/images-service.git#v2.3.4

The problems are:

1. We need to manage versions of subprojects in vagga, i.e. we need to reintroduce all the things git submodule can do
2. It's easy to forget committing right revision when subproject directory is overriden
3. There are security implications