middlewarehq
Allow Users to Sync Older Data
Why do we need this ?
- The current ETL mechanism uses DEFAULT_SYNC_DAYS var stored in the env of the image to sync data for the past DEFAULT_SYNC_DAYS at first time data sync.
- The use can update the DEFAULT_SYNC_DAYS in env in dev mode and run the server to sync past data but there is no mechanism to do so in the prod image.
Acceptance Criteria
- [ ] Add Settings for DEFAULT_SYNC_DAYS. Add enum, getter and setter.
- [ ] Update DEFAULT_SYNC_DAYS usage throughout the code base to use setting instead of env variable.
- [ ] Add BookmarkService: Add a core boomark service to fetch, set and reset all bookmarks
- [ ] Update usage of BookmarkService throughout the codebase
- [ ] Update BookmarkService.reset_bookmarks in DEFAULT_SYNC_DAYS setter to reset all bookmarks if default sync days is update or increased.
- [ ] Add UI for updating DEFAULT_SYNC_DAYS Setting.
Further Comments / References
Work Breakdown:
- Add BookmarkService - get_bookmark(entity_id, entity_type, provider)->datetime - set_bookmark(entity_id, entity_type, provider, bookmark_timestamp)->datetime - reset_org_bookmarks(org_id) - _get_repo_bookmark(repo_id) - _set_repo_bookmark(repo_id, datetime) - _get_workflow_bookmark(repo_id) - _set_workflow_bookmark(repo_id, datetime) - _get_incident_bookmark(service_id, provider, bookmark_type ) - _set_incident_bookmark(service_id, provider, bookmark_type, datetime) - _get_merge_to_deploy_bookmark(repo_id) - _set_merge_to_deploy_bookmark(repo_id, datetime) - _reset_repo_bookmarks(repo_id, datetime) - _reset_workflow_bookmarks(id, datetime) - _reset_incident_bookmarks(service_id, provider, bookmark_type, datetime) - _reset_merge_to_deploy_bookmarks(id, datetime
- Update usages of all getters and setters for bookmarks across the codebase
- Add a new setting for default sync days
- Update sync days usage across the code base to use setting instead of env vars
- Update setter for DEFAULT_SYNC_DAYS_SETTING to call reset_org_bookmarks
- Update setter for DEFAULT_SYNC_DAYS_SETTING to call reset_org_bookmarks
- Add UI for updating Default sync days setting
@adnanhashmi09 Are you working on this issue? I have also been working on this 🥲.
Ah dang!
You could pick up the frontend changes for this task. I have made a PR for the backend changes for DEFAULT_SYNC_DAYS
Hey @Kamlesh72 apologies, it would be ideal to get issue assigned before starting work. Been there, done that. Still remember I spent a whole week to open a PR to RocketChat that was closed by one of their engineers midway :")
We can place the "Update Sync Days" button inside "Settings" button of Dora Metrics page. Is this ok? @samad-yar-khan
@Kamlesh72 That could work and open a modal @jayantbh / @e-for-eshaan can hint better at the UI
Because this isn't strictly a Dora related setting, I recommend creating a new page for this, perhaps as the /settings path.
It won't have much today, but eventually it can have more things.
As of this comment the backend work for this has been done. In above PRs.
Additional context for UI and BFF:
Our sync depends on the default sync days value and if someone increases the default sync days setting, we should sync older data respecting the new default sync days for existing and prev data. The original plan was to call bookmark service from settings to reset the bookmark if the default sync days setting is updated. This side-effect would have been convenient but magic. So we decided to make an API in #514 and whenever the UI calls to update the default sync days setting, it should also sequentially call reset bookmark api. If the second api to reset bookmark fails, we need to reset the setting to previous value.
Is this ok? Any suggestions? I will later add toaster on this. The Sync Days and Bookmark reset are sequentially called.
https://github.com/user-attachments/assets/b489d62f-5125-4548-bab8-48b5ecf0ca74
For what purpose bookmark_timestamp is used in bookmark reset API?
So reset_bookmark will just reset all bookmarks to this param, but if its not passed, it will just pick it from the settings. So you can ignore the param for now.
We'll also need a toast to show the update is done and now we'll resync the repos.
As of this comment the backend work for this has been done. In above PRs.
Additional context for UI and BFF: Our sync depends on the default sync days value and if someone increases the default sync days setting, we should sync older data respecting the new default sync days for existing and prev data. The original plan was to call bookmark service from settings to reset the bookmark if the default sync days setting is updated. This side-effect would have been convenient but
magic. So we decided to make an API in #514 and whenever the UI calls to update the default sync days setting, it should also sequentially call reset bookmark api. If the second api to reset bookmark fails, we need to reset the setting to previous value.
Okay. change of plans. Doing this from bff was too much context sharing, so I opened a PR to do this in backend. #518
