marquez
marquez copied to clipboard
Marquez Dataset APIs don't return `outputFacets` or `inputFacets`
Some time ago, the OpenLineage spec was changed to include outputFacets
and inputFacets
in the Dataset
s reported by a RunEvent
. Several of the integrations, including the Spark integration and the GreatExpectations one, were changed to start reporting facets, such as OutputStatistics
and DataQuality
as either outputFacets
or inputFacets
.
In Marquez, the Dataset
APIs return facets for Datasets and DatasetVersions, but only read the facets
field of the Datasets reported from OpenLineage. This means the newer facets reported by the integrations are absent from the API response. We should update the queries to include those newer facets.