nautobot
[Idea] Replace monolithic Sync diff with per-record diffs
Environment
- Nautobot version: all
- nautobot-ssot version: all
Proposed Functionality
As noted in #144, #53, #140, etc., the current approach of storing the entire diff set for a given sync run in a single JSONField in a single Sync record doesn't scale well at all. To be able to usefully and performantly represent syncs involving 10k, 100k, or more changed objects, we should consider a different database representation for diffs.
The most "obvious" approach would be a single database record per changed object, with the UI taking responsibility for asynchronously fetching all relevant records associated with a given sync in order to provide a unified view of the synced changes as a whole.
I would say "just use core's ObjectChange model for this", but:
- that model stores a snapshot rather than diffs (at least for now)
- we need the ability to store "speculative" changes (i.e. dry run) in addition to "as-applied" changes.
So given that, we probably need a dedicated ObjectDiff model specifically in SSOT.
Use Case
Scalability and performance.
To add to this, I think it'd also be a good solution to the "caching" use case where we want to use previous data from a dryrun and execute on it without having to reprocess everything.
Any chance we can maybe use a foreign key to the object change model so we don't have to store everything twice? Perhaps the diff can be ad-hoc calculated from the object change records?
Although I guess this doesn't help with the speculative use case.
Using ObjectChange also wouldn't help with outbound syncs (Nautobot-to-other-system), but it's worth consideration anyways.
The non-Nautobot SoR use cases would be my primary reason for not wanting to have that direct link. Also, I don't think it's that easy to grep what change happened from those records. The current diff that's shown in SyncLogs is good but might be better to show the full before/after of the loaded object and potentially a link back to the previous change for that object so you could do a "git diff history" type tracking for objects?