metaflow
metaflow copied to clipboard
Netflix
:rocket: Build and manage real-life ML, AI, and data science projects with ease!
This PR adds more support for secure computing environments for metaflow in kubernetes and with argo. • Refactored so that common code is used to create the kubernetes client container...
For example, the following code seems not working: ```python class SomeFlow(FlowSpec): some_condition = Parameter( ... ) @step def start(self): if self.some_condition == ... : self.next(self.a) else: self.next(self.b) @step def a(self):...
- Add ` ARGO_EVENTS_SENSOR_NAMESPACE` to config (defaults to ` KUBERNETES_NAMESPACE` ) - Add ` sensor_namespace` parameter to argo client - Inititalize argo client with sensor namespace parameter when creating sensor...
This PR adds support for specifying limits (as opposed to requests) for workflows on Kubernetes. It addresses issue [1544](https://github.com/Netflix/metaflow/issues/1544). Notes: * I've added a single flag in the Metaflow config...
Tested running workflow using the following command `METAFLOW_ZIPKIN_ENDPOINT="{ENDPOINT_HERE}" METAFLOW_TRACING_URL_TEMPLATE="{END_POINT_HERE}/{trace_id}" python workflow.py run` Console result looks like the folloinwg: ``` 2023-11-13 20:36:10.220 Done! See the run in the UI at http://url/HelloFlow/10180...
I have a flow where I am accessing different attributes (numpy arrays in this case) depending on a variable defined by a `foreach` split. Upon joining back, `merge_artifacts` is failing...
output sent to stdout fails due to metaflow/plugins/datatools/s3/s3.py +1542 ``` for l in ok_lines: idx, rest = l.split(b" ", maxsplit=1) ``` It doesn't like the output starting ``` { "name":...
If METAFLOW_OTEL_ENDPOINT is set and neither METAFLOW_SERVICE_AUTH_KEY or METAFLOW_SERVICE_HEADERS are set, then you get a warning message `WARNING: no auth settings for Opentelemetry` followed by a load of errors as...
For [time scheduled argo workflows](https://docs.metaflow.org/production/scheduling-metaflow-flows/scheduling-with-argo-workflows#time-based-triggering), is it possible to read the timestamp the flow was scheduled for? For instance if a flow fails and i want to retry it, how...
### Summary Argo does not propagate the spec.podLabels to the pods. Only the sections under templates from the WorkflowTemplate are propagated to the Workflow and the pods. Their docs are...
