Add k8s executor [WIP]

[mattrasmus]

Hi @dakoner, I am using this branch to provide more detailed code improvements for #19. Please feel to incorporate this code. I'll continue to push to here over the coming weeks as we stabilize the branch.

When you get a chance, could you run make format on your branch to reduce the diff due to linting?

TODO

• [x] Make kubernetes an extra install.
• [x] Generalize file staging so that this can be used independent of aws (s3).
• [x] Pass secrets for object storage access (such as s3) using k8s secrets mechanism.

[mattrasmus]

@dakoner I self commented the changes I've made thus far. Feel free to incorporate these into #19.

[dakoner]

I hope to review this on friday.

[ricomnl]

@dakoner @mattrasmus this line needs to be changed to:

container = client.V1Container(name=name, image=image, args=command, env=env)

In kubernetes, the command overwrites Dockers ENTRYPOINT while args overwrites CMD. I ran into an issue with a micromamba image that activated the environment in the entrypoint.sh and therefore didn't work. Changing the command to args fixed this though. Source: https://stackoverflow.com/questions/59248318/kubectl-run-command-vs-arguments

[mattrasmus]

@dakoner @mattrasmus this line needs to be changed to:

container = client.V1Container(name=name, image=image, args=command, env=env)

Thanks @ricomnl for pointing this out! We'll incorporate.

[mattrasmus]

Thanks everyone! I'll merge this first beta version of k8s support as is now, and we can continue to refine/improve in smaller follow up PRs.