Xiangrui Meng

We have the same issue with Spark, as documented in https://issues.apache.org/jira/browse/SPARK-1835 .

@sonichi Thanks for the references! I will take a look at ML.NET. On the pipeline execution engine, I used bazel during prototyping. The pipeline engine won't be end-user facing (except...

@addisonklinke MLX and Kubeflow serve different personas. The initial target for MLX is data scientists who are new to ML under production settings, which we believe are the majority. IMHO...

> Are there any code examples? For notebooks (Jupyter and Databricks) and non-notebook IDE-based code. Do you mean working code examples or mocked? In the attached video, I did a...

> @mengxr I am also wondering how this be integrated with other pipelines such as Airflow, as part of the pipelines are using Scala-based spark jobs do the heavy lifting...

> I broadly enjoy the idea, and this is largely an MLFlow-native version of any other pipeline library that exists (i.e. most of these have converaged to the same broad...

> I like this direction, but would really want to see MLflow utilize or integrate with existing pipeline tools like Argo Workflows or Apache Airflow. Maybe take a look at...

@jnp Thanks for your feedback! > 1. Where will the pipeline templates be stored? It might make sense to store the templates with the mlflow service itself for management and...

@jhseu After we verify correctness, we can keep this PR open so less work for users who want to try out Spark 3.0 preview with spark-tensorflow-connector.

@WeichenXu123 Could you explain the test flakiness? Is it relevant to Spark 3.0 upgrade? If not, let's submit another PR so the fix can go in.