migrations: add a section around inconsistencies possible in LMS shadow modes

[ajstorm]

Adam Storm (ajstorm) commented:

We should add a section to the LMS documentation which talks about "data consistency when using the LMS". We allude to consistency issues at a very high level when talking about the different shadowing modes, but we never say anything like "if you use this mode, inconsistence between the two databases can occur in the following ways...".

For example, in sync mode, we can get inconsistencies due to the fact that the write fails to the non-source-of-truth and the failure is not reported. Even in strict sync mode, we can get inconsistencies because when errors occur on one side, we don't rollback the work on the side where it succeeded.

In general, we should call out all of the possible inconsistencies so that there are no surprises when the LMS is deployed.

Jira Issue: DOC-9597

[exalate-issue-sync[bot]]

Shannon Bradshaw (shannonbradshaw) commented: Adam Storm this and https://cockroachlabs.atlassian.net/browse/DOC-9598 look like they emerged either from internal discussion, a customer support case, or both. If so, would you please link relevant discussion or escalations here? I did look briefly in Slack and GitHub but couldn’t find anything.

[exalate-issue-sync[bot]]

Adam Storm (ajstorm) commented: https://cockroachlabs.slack.com/archives/C02TLESRWCB/p1706281467155219