conda list and clean commands

[mt-ob]

TODOs:

• [x] How to associate step names with environment IDs?
• [x] Where are dependency lock files located? Since we need to delete those as well..

[iamsgarg-ob]

LGTM.. just not sure how easy or hard it is to add tests.

[romain-intel]

haven't looked yet in detail but I think we should name it environment to keep consistency with the extension.

[savingoyal]

@romain-intel, this is more of a hidden utility to clean out the underlying conda environments than providing a top-level environment manipulation command. there is a good reason that we may want to have environment as yet another command in the near future.

[romain-intel]

I mean, it would be exposed to the user and there is similarity with this: https://github.com/Netflix/metaflow-nflx-extensions/blob/main/metaflow_extensions/netflix_ext/plugins/environment_cli.py so I wanted to try to minimize (more) divergence. There is also an argument for doing this in metaflow environment as opposed to myflow.py --environment=conda environment but that may be a larger conversation. If this is just something hidden, we should hide it better or provide a utility. Also, we could start by providing these commands and then expand to others. I just feel a bit uncomfortable providing a user-visible "hidden utility" like this I guess.

[mt-ob]

I remember asking @savingoyal if we wanna add this at the metaflow conda level or perhaps python flow.py conda level. The latter was probably chosen because we kind of operate on a per flow basis i.e. conda environments for each step etc. and we want the ability to list them + clean them etc.

At the metaflow conda level, it would be a bit difficult to get the ctx.obj I guess -- which allows all that functionality. I could be wrong though.

[romain-intel]

I remember asking @savingoyal if we wanna add this at the metaflow conda level or perhaps python flow.py conda level. The latter was probably chosen because we kind of operate on a per flow basis i.e. conda environments for each step etc. and we want the ability to list them + clean them etc.

At the metaflow conda level, it would be a bit difficult to get the ctx.obj I guess -- which allows all that functionality. I could be wrong though.

Totally agree that it doesn't necessarily have to be at the metaflow level. However, I would prefer it be named environment to be more compatible with the CLI in the extension (it's one or the other which makes more sense).