Lineage not getting displayed for all tables

[mithun1979]

Lineage became visible for a table on the first run. However, its no longer changing/updating after including additional notebooks tables. The code does a simple CTAS; CREATE TABLE <TABLE_NAME> USING DELTA AS SELECT * from <SOURCE_TABLE_NAME>

The source Table is in ADLS Gen2. The Target table is a managed table in DBFS (Databricks Default Database).

Expected behavior New Lineage information should show up in Purview Logs PurviewOut.log

OpenLineageIn.log

In PurviewOut.log, there is an error: Information 2023-05-24 10:00:39.049 Error Loading to Purview JSON Entiitesto Purview: Return Code: BadRequest - Reason:Bad Request Error 2023-05-24 10:00:39.049 Purview Publish Entity Metadata Error : Error :{"requestId":"fc68faa4-73c4-4808-a77b-2fe96f65546e","errorCode":"ATLAS-400-00-036","errorMessage":"invalid relationshipDef: process_dataset_outputs: end type 1: databricks_process, end type 2: databricks_notebook"} Error 2023-05-24 10:00:40.128 Executed 'Functions.PurviewOut' (Succeeded, Id=0783df86-0011-480e-90c2-1c3660514b4d, Duration=4766ms) Information

Screenshots NA

Desktop (please complete the following information):

• OS: Windows
• OpenLineage Version: openlineage-spark-0.18.0.jar
• Databricks Runtime Version: 11.3
• Cluster Type: Interactive
• Cluster Mode: No Isolation Shared
• Using Credential Passthrough: No

Additional context The Lineage data showed up the first time. So the setup seems to be good. It seems there is a ATLAS error in the PurviewOut.logs

[wjohnson]

@mithun1979 it looks like the input is okay but the output is pointing to /user/hive/warehouse/test_call_center_schema_chnaged and is not mapping correctly to hive metastore. It unfortunately is finding a databricks_notebook from the search results and trying to map the hive table to the first object that Purview search turns up as a match.

Support for Delta is limited but we will try to get better support for this.