metaflow icon indicating copy to clipboard operation
metaflow copied to clipboard
verified publisher icon Netflix

Rewrite IncludeFile implementation

Open romain-intel opened this issue 3 years ago • 2 comments

This rewrite addresses several issues:

  • IncludeFile were not properly returned in dump and via the client
  • Improves logic in create/trigger for step-functions for example
  • Properly consider the size of the included file when dumping artifacts
  • default value for IncludeFile can also now be a function

The code has been cleaned up overall as well and many more comments added.

romain-intel avatar Aug 11 '22 09:08 romain-intel

Testing[221] @ 00494f307774fe28fe20f0db1cddfeccf60b3e75

nflx-mf-bot avatar Aug 18 '22 09:08 nflx-mf-bot

Testing[221] @ 00494f307774fe28fe20f0db1cddfeccf60b3e75 PASSED

nflx-mf-bot avatar Aug 18 '22 11:08 nflx-mf-bot

Please merge #1155 before rebasing.

valayDave avatar Sep 30 '22 20:09 valayDave

A quick note on sequencing this PR - we can merge the GCP datastore implementation first, then fast-follow with the changes needed by this PR for Azure and GCP, and then merge this PR. wdyt @romain-intel ?

savingoyal avatar Oct 11 '22 07:10 savingoyal

This has been hanging a while. I would rather we merge this and then GCP. I also want to discuss the "let's move things to plugins" before we continue merging plugins (I keep bringing it up but it keeps getting delayed so we should discuss it).

romain-intel avatar Oct 11 '22 07:10 romain-intel