marquez icon indicating copy to clipboard operation
marquez copied to clipboard

Marquez Dataset APIs don't return `outputFacets` or `inputFacets`

Open collado-mike opened this issue 3 years ago • 0 comments

Some time ago, the OpenLineage spec was changed to include outputFacets and inputFacets in the Datasets reported by a RunEvent. Several of the integrations, including the Spark integration and the GreatExpectations one, were changed to start reporting facets, such as OutputStatistics and DataQuality as either outputFacets or inputFacets.

In Marquez, the Dataset APIs return facets for Datasets and DatasetVersions, but only read the facets field of the Datasets reported from OpenLineage. This means the newer facets reported by the integrations are absent from the API response. We should update the queries to include those newer facets.

collado-mike avatar Nov 04 '21 22:11 collado-mike