Meta42
Meta42
I'll follow up
> cc @iodone @turboFei Is there more detailed information to help me reproduce, I did not reproduce it locally.
@ulysses-you I refactored some code，could you help me to review it? Thanks
> I suggest this feature goes to extensions. The original design was implemented in the extension layer, extending QueryExecutionListener, so we can take this approach, right?
Two approaches are currently implemented： 1. Spark SQL Engine , `set kyuubi.operation.lineage.enabled = true` to enable lineage information logging. The lineage information will be added to `SparkOperationEvent` and output to...
> Let's clean the way 1 first, I think 2 is better and can still be improved In some scenarios the user submits SQL only to get the lineage information...
> @ulysses-you can't the current event framework achieve such a goal? with custom event and costume event handler? ```scala class SparkHistoryLoggingEventHandler(sc: SparkContext) extends EventHandler[KyuubiEvent] { override def apply(event: KyuubiEvent): Unit...
> @iodone can you create a issue that support track lineage from subquery and v2write ? we can improve that in future OK
This is not a bug in Kyuubi. Using "spark.jar=xxx.jar" which does not add the jar path to` spark.sharedState.jarClassLoader`, you need to use the` extraClassPath` conf to add the corresponding classpath....
> I think we'd better to define lineage clearly. The current lineage is easy to follow that we only consider plan's output, which means if a column is used to...