access to `dolt_branches` is not safe in multi-tenant environment

[zachmu]

Queries like select * from dolt_branches can fail under concurrent load when other clients delete a branch. This is because dolt_branches and some other system tables don't currently respect transactional isolation and always read from the current tip of the entire database.

Expectation is that branch information is managed transactionally, the same as table data.

First milestone is read isolation: dolt_branches should have the same contents as it did at transaction start throughout a transaction, regardless of changes made by other clients.

Second milestone is write isolation: prevent conflicting writes to a particular branch by multiple clients. We currently are only principled about moving branch heads transactionally for the dolt_commit() procedure. We need to bring in dolt_branch(), dolt_reset() and other operations that can move or delete a branch head.

[zachmu]

First milestone is released: reads of dolt_branches get their data from the database's state at transaction start time.

Writes are still not safe, can concurrent calls to dolt_branch can fail in strange ways.

[nicktobey]

"First milestone is released: reads of dolt_branches get their data from the database's state at transaction start time."

I'm noticing behavior that seems odd (to me), and I want to make sure I'm understand this correctly and whether it's related to this.

I connect to a local db with two dolt sql commands (so no server is running, two dolt clients spin up their own engines, but only one has a lock)

On the first client I create a branch.

On the second client I run select * from dolt_branches

The newly created branch does not appear in the results.

Reads to dolt_branches use data from the start of the transaction, so I make a new transaction in the second client by running:

start transaction;
commit;

These both succeed despite the client not having a lock on the db (because no changes are actually made, I guess.)

I run select * from dolt_branches again. The newly created branch still doesn't appear in the results. I kill the process and start a new one. This time, when I run select * from dolt_branches, I see the new branch.

Is this the expected behavior here?