Missing objects and object refs for contents

[en0ma]

Describe the bug While investigating the commp issue, I noticed content with id 30545853 in Estuary is missing its objects and object_refs in Estuary and in its shuttle. I suspect this kind of issue must have happened due to downtime on primary when content is being created or something.

Expected behavior The Content should have its object and object_refs database records in both primary and shuttle.

Actual behavior The Content is missing its object and object_refs database records in both primary and shuttle.

• [x] find and fix the bug that causes this
• [ ] A backfill/verification tool should be written to detect and backfill such contents.

[anjor]

On the verification side, are you thinking a tool that goes through the database, and finds any consistencies?

So potentially, we can configure some "invariants" that the data model should respect, and this tool checks whether any of those invariants are broken and flags them.

Fixing these inconsistencies would probably look like (1) backfilling if we can or (2) dropping the rows where it is not possible to backfill -- this will involve data loss.

Is my understanding roughly correct?

[en0ma]

On the verification side, are you thinking a tool that goes through the database, and finds any consistencies?

So potentially, we can configure some "invariants" that the data model should respect, and this tool checks whether any of those invariants are broken and flags them.

Fixing these inconsistencies would probably look like (1) backfilling if we can or (2) dropping the rows where it is not possible to backfill -- this will involve data loss.

Is my understanding roughly correct?

yes @anjor