Purview-ADB-Lineage-Solution-Accelerator
Purview-ADB-Lineage-Solution-Accelerator copied to clipboard
Support Reducing Mount Points in OpenLineage Payload Prior to Sending to Event Hubs
Describe the feature There may be cases where there are a large number of mount points defined and returned in the OpenLineage payload. As a result, we run into the issue where the Event Hub cannot handle the payload and the existing methods to reduce the payload size do not cover Mount Points
Suggested Implementation
- In the Ol-Message Consolidation, we could look at whether the mount points are used and which mount points are used.
- The risk is that if we get a complete and subsequent messages need the mount point, we may have discarded it already?