datahub
datahub copied to clipboard
feat(ui) Retrieve last ingested timestamp and display on frontend
This WIP PR gets the timestamp for when a dataset entity was last ingested by checking all of its aspects for the most recent lastObserved
in its systemMetadata
where the runId
exists and is not the default (no-run-id-provided
) which means that ingestion is what last touched this aspect.
I wanted to get a PR up quick first to get some feedback. I'm planning on extending this to all entity types and writing tests for the getLastIngested method.
Here's what it's looking like on the frontend:
within 1 week of ingestion, green
on hover
Orange when it's more than a week but less than a month
Red when it's more than a month since
Checklist
- [ ] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
- [ ] Links to related issues (if applicable)
- [ ] Tests for the changes have been added/updated (if applicable)
- [ ] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
- [ ] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub
Unit Test Results (build & test)
502 tests 502 :heavy_check_mark: 9m 5s :stopwatch: 116 suites 0 :zzz: 116 files 0 :x:
Results for commit 574de9e4.
:recycle: This comment has been updated with latest results.
Sorry for reviving this merged PR, @chriscollins3456 , but can I confirm if the logic considers TimeSeriesAspects as well? that is, if i programmatically ONLY form a time series aspect containing datasetStats and include a runId and lastObserved in the MCP, the last synchronized timestamp will also be updated?