canonical
Support TLS client session cache
@cole-miller pointed me to https://pkg.go.dev/crypto/tls#ClientSessionCache which might be useful in reducing the TLS connection overhead that contributes to the go-dqlite performance issues that we see.
As such, this introduces an LRU cache of size 64, which was chosen as it's the default size if 0 is passed instead, when initializing the cache.
I haven't tested its efficacy yet, so it's just a draft for now.
@cole-miller pointed me to https://pkg.go.dev/crypto/tls#ClientSessionCache which might be useful in reducing the TLS connection overhead that contributes to the
go-dqliteperformance issues that we see.
Speeding up the TLS session establishment is great but could we instead have permanent connections to the DB?
@cole-miller pointed me to https://pkg.go.dev/crypto/tls#ClientSessionCache which might be useful in reducing the TLS connection overhead that contributes to the
go-dqliteperformance issues that we see.Speeding up the TLS session establishment is great but could we instead have permanent connections to the DB?
I'm not entirely sure how easy that would be. I'm just spitballing, but I don't know if the dialer is smart enough to gracefully handle intermittent connection issues. I'd assume there's some sort of buffer period where we try to reuse the connection before establishing a new one, which means under conditions with poor connectivity, we would be waiting the duration of the buffer quite frequently, and opening a new connection anyway. We can get around that by just always eagerly opening a new connection.
I don't believe LXD uses persistent connections for dqlite either, though I'm not sure if that's the reason or something else.
I don't believe LXD uses persistent connections for dqlite either, though I'm not sure if that's the reason or something else.
I have the impression is that it does, it has the concept of a gateway which is brought up and tunnels dqlite's unix socket over it between members.
LXD separately does open a new TCP connection for its app level heartbeats though.
How does the go-dqlite App do it?
I don't believe LXD uses persistent connections for dqlite either, though I'm not sure if that's the reason or something else.
I have the impression is that it does, it has the concept of a gateway which is brought up and tunnels dqlite's unix socket over it between members.
A permanent connection is IMHO the way to go as otherwise your maximum requests per seconds goes down the drain. Inter cluster member latency is non negligible even inside a DC. A quick test between two machines connected to the same switch gives me ~1ms of RTT. So even if we make the TLS handshake mostly "free", paying for TCP establishment is leaving performance on the table.
LXD separately does open a new TCP connection for its app level heartbeats though.
Liveness checks are a different beast but even then, a persistent connection where a ping/pong exchange happens would IMHO be optimal in terms of resource usage. That bit is not on the critical performance path though.
I don't believe LXD uses persistent connections for dqlite either, though I'm not sure if that's the reason or something else.
I have the impression is that it does, it has the concept of a gateway which is brought up and tunnels dqlite's unix socket over it between members.
LXD separately does open a new TCP connection for its app level heartbeats though.
How does the go-dqlite App do it?
Yeah I was referring to the app-level heartbeats and leader detection. go-dqlite's implementation is very similar to LXD when using an external connection. The parts where it differs is the built-in heartbeat for role turnover, which opens new connections when searching for the dqlite leader.
