metaflow icon indicating copy to clipboard operation
metaflow copied to clipboard

feature: on demand docker environments

Open saikonen opened this issue 1 year ago • 3 comments

Adds support for on demand Docker images. Open for discussion

open todo's

  • [x] add fallback to conda environment when executing locally with --environment docker
  • [x] add support for imagebakery endpoint auth
  • [x] cache bakery image tags locally to avoid environment drift with consecutive deployments when using loosely pinned versions

saikonen avatar Apr 10 '24 14:04 saikonen

python for.py --environment=docker run without @kubernetes immediately returns an error that Image Bakery is not configured. Perhaps it shouldn't throw the error?

savingoyal avatar May 02 '24 15:05 savingoyal

the changes allow an extension to implement

def conda_supported_virtual_envs():
    return ["docker"]

in org_ext/config/mfextinit_org_ext.py to support --environment docker for @pypi and @conda


saikonen avatar Jun 26 '24 12:06 saikonen

also possible to fallback to a Conda environment for steps that can not be handled by another env implementation:

def init_environment(self, echo):
    self.delegate = CondaEnvironment(self.flow)
    ...
    if self.steps_to_delegate:
        self.delegate.validate_environment(echo, self.datastore_type)
        self.delegate.init_environment(echo, self.steps_to_delegate) # skips steps that were already handled.

saikonen avatar Jun 26 '24 12:06 saikonen

closing this in favour of developing the feature as an extension now that #1902 has been merged.

saikonen avatar Jul 18 '24 13:07 saikonen