Bigtable Database integrity error: Expected column: connected_at

[data-sync-user]

Yesterday we began seeing new Database error: Database integrity error: Expected column: connected_at errors reported in sentry:

https://mozilla.sentry.io/issues/4517044823/events/b866d03114874e1199c4ca2d5e0d4525/

This was previously seen and resolved by https://mozilla-hub.atlassian.net/browse/SYNC-4159, so we’re seeing a different case of this here.

It’s not clear what could produce a router row record that would produce this. Possibly a remove_channel call on such a ‘chid-only’ record could produce a record similar to a ‘chid-only’ but also including a version cell. However that seems a bit unlikely to have actually occurred.

Let’s double check what kind of row we’re dealing with by emitting the entire invalid row in the sentry event. This might require at least a partial fix for https://mozilla-hub.atlassian.net/browse/SYNC-4145.

┆Issue is synchronized with this Jira Bug