Running containers instead of bare commands

[ChristianKniep]

Wondering if you ever thought about using a container for execution instead of a process..

Should be rather simple to implement.

[yottahmd]

Thanks for raising the issue! We will prepare images and DockerCompose setup later. As of now, there's only a Dockerfile for dagu server here.

[ChristianKniep]

Hey, my goal is not to run the server in a container (even though that is certainly a good thing to do anyways), but instead run the tasks containerized. Like

steps:
  - name: step 1
    container: alpine:3.16.2
    command: echo hello
  - name: step 2
    container: pandoc/core 
    command: README.md -o outfile.pdf
    depends:
      - step 1

That way you abstract away the need to have all the tools installed (in a specific version).

[yottahmd]

Ah, sorry for my misunderstanding. Being able to run containers natively is a great idea. Let's look into how to implement such a feature.

[ChristianKniep]

I reckon it should be rather simple to create a ContainerExecution.go extension. https://github.com/yohamta/dagu/blob/main/internal/executor/command.go

I'll try to give it a shot some time. Need to prep for a workshop tho. :)

[yottahmd]

That would be a huge help! Thank you very much :)

[Arvintian]

I think we can just implement the executor interface https://github.com/yohamta/dagu/blob/main/internal/executor/executor.go#L12. By the way, i think we can support docker container or kubernetes pod executor (confilct with kubernetes's cronjob?). About more the executor may be need settings info. We should open a field to config executor?

[yottahmd]

Thank you! I agree with you. I am not sure but think we will need to add some DAG fields to config executor for docker / k8s.

[yottahmd]

~~Let's extend the exectutor interface first~~, and then add the Docker container executor.

[yottahmd]

Added executorConfig field in Step struct in #321 PR. It turns out we didn't have to update the Executor interface thanks to the great design done by @Arvintian Now I think we can add any executor as well as docker container executor, by adding a new Executor as HTTPExecutor: https://github.com/yohamta/dagu/blob/main/internal/executor/http.go#L16

[yottahmd]

I think we can use DockerEngine's Go SDK: https://docs.docker.com/engine/api/sdk/

[yottahmd]

@ChristianKniep Sorry to keep you waiting :) Are you still interested in making a simple ContainerExecution.go extension?

[ChristianKniep]

@yohamta I am on the road attending a workshop, so I was busy and will be back next week. I still super interested in a container executor. If you are faster than me - I would not mind.

[ChristianKniep]

If you have some boilerplate for me to fill out I can add the container logic.

[yottahmd]

@ChristianKniep That's great. No need to rush, thank you so much. Good luck with the workshop 👍 Will prepare the boilerplate later :)

[yottahmd]

@ChristianKniep Here is the boilerplate and an example DAG. You can run the example DAG with this command:

go run ./cmd start ./examples/docker-container.yaml

Thanks!

[ChristianKniep]

Cool, some questions.

• Do you expect the process to have access to files? Like: A container instance might create a volume for each step and allow those to be mounted by subsequent steps to access data.
• all parents of a step needs to finish before the next step is allowed to start, right?

[yottahmd]

Do you expect the process to have access to files? Like: A container instance might create a volume for each step and allow those to be mounted by subsequent steps to access data.

I think it would be better to leave that to the user's configuration. I thought it would be better to be able to set Binds argument in the DAG's executorConfig. If the user wants to mount it on a local volume, it could be configured as follows (either bind mount or volume).

steps:
  - name: step1
    executor: docker
    executorConfig:
      - image: node
        binds:
          - "/localPath:/pathInDocker"
    command: npm install

Then the DockerConfig struct in the boilerplate I prepared would need to be modified so that it can have Binds key. Binds is the field of the HostConfig struct in Docker's Go SDK.

all parents of a step needs to finish before the next step is allowed to start, right?

I thought it depends on the setting of the depends key in the DAG, if depends key has the name of the parent step, then that parent step will complete first.

steps:
  - name: parent
    executor: docker
    executorConfig:
      - image: node
        binds:
          - "/localPath:/pathInDocker"
    command: npm install
  - name: child
    executor: docker
    executorConfig:
      - image: node
        binds:
          - "/localPath:/pathInDocker"
    command: npm start
    depends:
       - parent

The DAG definition above looks redundant, but I think that can be improved separately by adding some generic template feature or something where the user can reuse common settings for the container steps. Is this answering your question or any different ideas?

[ChristianKniep]

ok, makes sense... Lemme give it more thought in the next couple of days.

[yottahmd]

no problem :)

[yottahmd]

It is implemented in v1.9.2.

Explanation about Docker Executor

[ChristianKniep]

Awesome @yohamta, thanks for implementing this and sorry that I did not.

[yottahmd]

@ChristianKniep Don't need to be sorry! Thanks for giving us the fantastic idea :)