metaflow
metaflow copied to clipboard
Netflix
Package code in more cases
Currently, the code is packaged for Batch runs only. To be able to fully replicate and version things, it would be nicer if code was always stored as an artifact of each step.
We are also missing this feature. We regularly make runs in local mode and would like to store our code as an artifact. Are there any plans to address this feature?
I mentioned in the community conversation (https://gitter.im/metaflow_org/community) that it would be super valuable to be able to rerun a previously successful Run, but with different Parameter(s) or different input data. Basically the resume command but on successful runs with the ability to change Parameter values
@tuulos mentioned that there is effort on providing a top-level command for bootstrapping a project based on an existing run. This would be very useful for us
Our particular use case right now for metaflow is to use it in creating datasets with proprietary feature engineering, and also have the ability to "replay" that same data set with additional/different input data points at any time in the future. Since our feature engineering logic may change over time, its important we can reproduce the same features from a previous run with new data
another +1 here https://gitter.im/metaflow_org/community?at=6075e54e06e2e024e84f645c
+1. Teams we're speaking with want to pack all code into the docker images and avoid downloading it from S3. "Everything in docker images" is aligned with corp. rules (central vulnerabilities scanning, licenses check, etc.) and becoming a problem otherwise. Edit: With code packages generated locally, it's should be easier to pack them into dockers.
