oso icon indicating copy to clipboard operation
oso copied to clipboard

Data completeness dashboard

Open ryscheng opened this issue 1 year ago • 5 comments

Describe the feature you'd like to request

Would be nice to have a status page on the web app that allows us to easily visualize what data is missing

Describe the solution you'd like

I think we can just visualize the event pointer table, looking for artifacts where the end date is older than a certain buffer from the current date.

Open question - how do we know if a start date is insufficiently early? CC @ravenac95

Describe alternatives you've considered

TBD

ryscheng avatar Nov 20 '23 17:11 ryscheng

Can we just put this in the project page so people can clearly see what data is missing?

ryscheng-mobile avatar Dec 12 '23 13:12 ryscheng-mobile

I'm leaving this here but it's possible that with the tools we are intending on using, dagster/airbyte we may already get a UI to see the status of all data ingestion.

ravenac95 avatar Jan 10 '24 17:01 ravenac95

When was this added to sprint 65?

ravenac95 avatar Apr 17 '24 16:04 ravenac95

Looks like we're on track to using Dagster for this purpose once we get some public instances running in our k8s cluster

ryscheng avatar May 05 '24 17:05 ryscheng

Lets repurpose this as populating metadata in MaterializedResult for all assets

ryscheng avatar May 21 '24 17:05 ryscheng

Also closed with #1633

ravenac95 avatar Jun 17 '24 13:06 ravenac95