Purview-ADB-Lineage-Solution-Accelerator
Purview-ADB-Lineage-Solution-Accelerator copied to clipboard
microsoft
Databricks python activity lineage issue
Describe the bug We have some ADF orchectrated python activity jobs which trigger a python file in dbfs.
Lineage for the same is not getting reflected in purview. I can see some events related to that is coming to purview out function with some errors.
To Reproduce Steps to reproduce the behavior:
- Create a python activity job from ADF .
- Observe the lineage
Logs
{"_producer":"https://github.com/OpenLineage/OpenLineage/tree/0.18.0/integration/spark","_schemaURL":"https://openlineage.io/spec/1-0-5/OpenLineage.json#/$defs/RunFacet","spark-version":"3.0.1","openlineage-spark-version":"0.18.0"}}},"job":{"namespace":"adb-2636379187030124.4#default","name":"adb-2636379187030124_4_azuredatabricks_net_execute_create_data_source_table_as_select_command_smds_C_ALT_CODE","facets":{}},"inputs":[{"namespace":"dbfs","name":"/mnt/datalake-prod/raw/smds/initial_reload/MDM_INFM_SMDS.C_ALT_CODE","facets":{"dataSource":{"_producer":"https://github.com/OpenLineage/OpenLineage/tree/0.18.0/integration/spark","_schemaURL":"https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet","name":"dbfs","uri":"dbfs"},"schema":{"_producer":"https://github.com/OpenLineage/OpenLineage/tree/0.18.0/integration/spark","_schemaURL":"https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet","fields":[{"name":"ROWID_OBJECT","type":"string"},{"name":"CREATOR","type":"string"},{"name":"CREATE_DATE","type":"string"},{"name":"UPDATED_BY","type":"string"},{"name":"LAST_UPDATE_DATE","type":"string"},{"name":"CONSOLIDATION_IND","type":"string"},{"name":"DELETED_IND","type":"string"},{"name":"DELETED_BY","type":"string"},{"name":"DELETED_D 2023-02-28T05:18:29Z [Information] Executed 'Functions.PurviewOut' (Succeeded, Id=cb9445d3-a201-4f65-b2e3-c54fcdcb51c7, Duration=117ms) 2023-02-28T05:18:29Z [Error] AdbClient-GetSingleAdbJobAsync: error, message: Unexpected character encountered while parsing value: {. Path 'tasks[0].libraries[0].pypi', line 1, position 1516. 2023-02-28T05:18:32Z [Information] Executing 'Functions.PurviewOut' (Reason='(null)', Id=181ec998-7e09-4ba1-ad01-b5474fdb9e6e) 2023-02-28T05:18:32Z [Information] Trigger Details: PartionId: 0, Offset: 124585262552, EnqueueTimeUtc: 2023-02-28T05:18:31.5100000Z, SequenceNumber: 3045, Count: 1 2023-02-28T05:18:32Z [Verbose] Sending invocation id:181ec998-7e09-4ba1-ad01-b5474fdb9e6e 2023-02-28T05:18:32Z [Verbose] Posting invocation id:181ec998-7e09-4ba1-ad01-b5474fdb9e6e on workerId:a70f5206-8b51-40ea-9b85-2916ca30d772
Screenshots If applicable, add screenshots to help explain your problem.
Python file location is in dbfs
Hi, @Kishor-Radhakrishnan thank you for bringing this to our attention!
Would you please try the hotfix/librariesInTasks.
With the fix, I was able to create a Python File job in ADF and have it run on a cluster configured to work with the solution accelerator and then observe lineage in the Microsoft Purview UI.
@wjohnson Thanks for the quick fix . I deployed the fix and I can see lineage for one of our dev flow.
Will do further validations.
Thank you so much Kishor! We've merged this into main and will get this into the release branch as soon as we get the last few issues you've reported.