Replication variables too sensitive to definition order

[zachmu]

Setting up a new read replica, you have to define the variables in a very particular order.

This fails:

dolt sql -q "SET @@persist.dolt_read_replica_remote = origin;"
dolt sql -q "set @@persist.dolt_replicate_all_heads = 1;"
dolt sql -q "set @@persist.dolt_replication_remote_url_template = 'gs://dolt-bucket/{database}';"

error on line 1 for query set @@persist.dolt_replicate_all_heads = 1: replication failed: invalid replicate heads setting: dolt_replicate_heads not set

This works:

dolt sql -q "set @@persist.dolt_replicate_all_heads = 1;"
dolt sql -q "set @@persist.dolt_replication_remote_url_template = 'gs://dolt-bucket/{database}';"
dolt sql -q "SET @@persist.dolt_read_replica_remote = origin;"

In general this just needs to be far more forgiving. The default behavior right now is to prevent the server from running (including any dolt sql commands) if the replication configuration has an issue, and this is wrong. We should log warnings and continue by default.

[max-hoffman]

dolt_skip_replication_errors could be default true?

[zachmu]

That would fix the immediate issue, yes.

[dphil]

I believe this is related to an issue I'm seeing.

On a db that was intended to be a read replica, I ran the following without issue: dolt sql -q "set @@persist.dolt_read_replica_remote = 'origin'"

Then, I ran the following, which included a typo in the var name (though, Dolt seemed to understand it anyway...?):

$ dolt sql -q "set @@persist.dolt_read_replicate_heads = 'main'"
error on line 1 for query set @@persist.dolt_read_replicate_heads = 'main': replication failed: invalid replicate heads setting: dolt_replicate_heads not set
replication failed: invalid replicate heads setting: dolt_replicate_heads not set

Then I fixed the typo and ran:

$ dolt sql -q "set @@persist.dolt_replicate_heads = 'main'"
error on line 1 for query set @@persist.dolt_replicate_heads = 'main': replication failed: invalid replicate heads setting: dolt_replicate_heads not set
replication failed: invalid replicate heads setting: dolt_replicate_heads not set

Every attempt to set any variable after this produced the exact same error message. In fact, even just querying the variables produced this error:

$ dolt sql -q "show variables like 'dolt%'"
error on line 1 for query show variables like 'dolt%': replication failed: invalid replicate heads setting: dolt_replicate_heads not set
replication failed: invalid replicate heads setting: dolt_replicate_heads not set

So it seems something got into a bad state.

[dphil]

I deleted and re-cloned the repo, and then set the vars in the "proper" order (dolt_replicate_all_heads, then dolt_read_replica_remote) and there were no issues this time.

[max-hoffman]

Yes this is the same issue. A couple of options:

1. Using dolt config --add --local sqlserver.global.dolt_replicate_heads 'main' would bypass the SQL engine to fix the variable. This is a temporary escape hatch, we still need to fix the core design flaw.

2. dolt_skip_replication_errors is currently set to false by default, but if set to true will log the error and let you precede with fixing the config. dolt_skip_replication_errors can be set with the dolt config command as well.

[timsehn]

Resolving in favor of: #6019