Romain

I think we definitely need a story around this. Based on the thread and other conversations had, I think there are a few concerns to address: - overall size of...

@talebzeghmi, do you mean a TTL on each run and having things auto-clean up after a while? It could be possible to basically run the GC job after each flow...

So, the code you refer to is this: ``` DATATOOLS_S3ROOT = from_conf( "METAFLOW_DATATOOLS_S3ROOT", os.path.join(from_conf("METAFLOW_DATASTORE_SYSROOT_S3"), DATATOOLS_SUFFIX) if from_conf("METAFLOW_DATASTORE_SYSROOT_S3") else None, ) ``` and it does seem to be conflicting with https://github.com/Netflix/metaflow/blob/master/metaflow/main_cli.py#L490...

I like this proposal too but have a few questions: - what is the priority order, I would see this but not sure if you are thinking the same: global...

@savingoyal , yes, sorry, I listed the list in the reverse order (ie: global image default overridden by flow specification overridden by etc) so I think we are saying the...

I would agree with that. i would also list the various priority and ordering (even if that is not 100% due to this proposal) so that it is clear in...

I see your point @oavdeev and I do see that `@image` specifies dependencies and `@resources` is more of a performance thing but one could also argue that if you specify...

The namespace is implicitly the user (which I thought was the initial intent) but yes, this is exactly the type of conversation I was mentioning we should have around this...

Testing is fine now on my end. The one failure is transient. Small comments remain and we can then go ahead and merge.

NOTE: Not compatible with include file changes. This PR can be made compatible fairly simply by adding the relevant methods in the includefile_support.py file.