go-dqlite
go-dqlite copied to clipboard
canonical
Parallellize connectAttemptAll
Parallellize https://github.com/canonical/go-dqlite/blob/88e5f725af8b2dd82d353d82e28fa78deb0be8ad/internal/protocol/connector.go#L117
instead of trying the nodes in sequence.
Currently we try to connect to one server at a time until we connect to the leader or until a server tells us who the leader and we try that server next. This strategy has caused some issues in the past in LXD where the whole time budget could be spent attempting to connect to a node that is not responding. I suspect it might also cause issues in microk8s that has stricter timing constraints than LXD.
Instead of that strategy, I would attempt to connect to all servers in parallel in a goroutine, if there is a leader, it should be one of those and we should be able to connect to it. If a server would report that a certain server is the leader and it's not in the list of the servers we are attempting to connect to, I would also attempt to connect to that server.
@stgraber I remember you saying that there are LXD clusters with 80+ nodes, attempting to connect to all 80 at the same time might be a bit too optimistic? @freeekanayaka Any obvious issues with this approach?
Just a couple of notes:
- Adjusting time budgets could be a good first move: for example the connect (or RPC, once connected) timeout could be made low enough that an unresponsive node won't create significant hiccups
- It seems that we currently attempt all servers found in the store, regardless of their role, probably they should be sorted with voters first, standbys next and spare last, since the leader is highly likely to be found among voters
That might be enough to solve the issues being observed. If that's still not enough, then perhaps you could indeed try adding some degree of parallelism, e.g. attempt to connect to 3 or 5 nodes in parallel at a time (following the role-based servers ordering described above).
Regarding the roles, I believe the app package does propagate them correctly (see here), but I'm not so sure about LXD, that might need to be fixed.
Yeah, if we do parallel connections, we should do them in batches:
- All voters
- All standby
- Spares in batch of 3
In 99% of cases this would then limit it to just 3 parallel connections and the first batch should either hit the leader or be pointed to it pretty quick.
@vianaddsv not at present; we'd be happy to review a PR if this is something you need in the near term for your use-case.
