Proposal: "Context"-aware resources

[dnephin]

There are two conditions which result in sub-optimal behaviour:

• an env resource can modify the behaviour of a task, which requires all subsequent tasks be marked as "required" (they can't be skipped)
• compose resources (and the proposed service and job:serve resources) create long running processes, so they must always run. There's no way to skip these tasks without doing some kind of look-ahead, but there isn't current enough information in the config format to determine how far to look ahead. It's not clear which of the following steps use the service.

I think both of these issues could be addressed by adding some form of a "context" (inspired by python context managers).

This config construct would provide a scope for the above resources, making it possible to skip long-running tasks (if all the tasks inside the scope are fresh), and only "require" tasks within the environment scope.

[dnephin]

I don't have any good ideas for how to model this in the config just yet.

[cescoferraro]

What if a job could take multiple commands? Analogous to a kubernetes pod, that aggregate multiple containers. The same job could take multiple commands that would share the same environment variables. Plus the image resource would not need to depend on env resources. Then deprecate the env resource and rename job to env

env=dev:
    net-mode: host
    user: "{user.uid}"
    use: dev-image
    mounts: [source]
    commands: 
        - name: webpack
          command: webpack --config build/prod.js
        - name: watch
          interactive: true 
          command: node src/hot.js
          depends: [webpack]
    vars:
      - "NODE_ENV=development"

Then call it like

$ dobi dev:go
$ dobi dev:webpack
$ dobi dev:hot

[dnephin]

What if a job could take multiple commands?

A container can only run one process (unless you exec). Would each command be an exec, or would they be different containers? Are they sequential or parallel?

Instead of answering all of those questions, why not just add a bash script to run the commands the way you want to? Or separate jobs of they are meant to be independent.

The same job could take multiple commands that would share the same environment variables.

I think the problem is more with the non-job resources. You may want to share a captured environment variable with an image:build or image:tag. That doesn't work as a multiple-command job.

Then deprecate the env resource and rename job to env

Not sure how this would work. They are very different right now. env populates environment variables, and job runs containers. Merging them seems like it would be overloading the concepts.

[dnephin]

I'm thinking something like this:

A new field requires available on most resources that only allows "context-like" tasks (currently job:capture, compose:run, and job:serve). These "context-like" tasks would not be allowed in depends any more.

job=export-db:
  ...
  requires: [empty-db]

When the tasks run, any of these "context-like" tasks are skipped until they are actually required.

The ExecContext will need to track which tasks have been required to it can defer their running.