datahub icon indicating copy to clipboard operation
datahub copied to clipboard

feat(ui) Retrieve last ingested timestamp and display on frontend

Open chriscollins3456 opened this issue 2 years ago • 1 comments

This WIP PR gets the timestamp for when a dataset entity was last ingested by checking all of its aspects for the most recent lastObserved in its systemMetadata where the runId exists and is not the default (no-run-id-provided) which means that ingestion is what last touched this aspect.

I wanted to get a PR up quick first to get some feedback. I'm planning on extending this to all entity types and writing tests for the getLastIngested method.

Here's what it's looking like on the frontend:

within 1 week of ingestion, green image

on hover image

Orange when it's more than a week but less than a month image

Red when it's more than a month since image

Checklist

  • [ ] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • [ ] Links to related issues (if applicable)
  • [ ] Tests for the changes have been added/updated (if applicable)
  • [ ] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • [ ] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

chriscollins3456 avatar Aug 09 '22 20:08 chriscollins3456

Unit Test Results (build & test)

502 tests   502 :heavy_check_mark:  9m 5s :stopwatch: 116 suites      0 :zzz: 116 files        0 :x:

Results for commit 574de9e4.

:recycle: This comment has been updated with latest results.

github-actions[bot] avatar Aug 09 '22 20:08 github-actions[bot]

Sorry for reviving this merged PR, @chriscollins3456 , but can I confirm if the logic considers TimeSeriesAspects as well? that is, if i programmatically ONLY form a time series aspect containing datasetStats and include a runId and lastObserved in the MCP, the last synchronized timestamp will also be updated?

xiphl avatar Dec 29 '22 03:12 xiphl