gocql
gocql copied to clipboard
Make tracing requests shard aware
Currently, all tracing requests use a single connection, the control connection; this connection might or might not be "the right" connection (in terms of which shard it's connected to) for the given session_id (which is the partition key in system_traces.sessions and system_traces.events).
Relevant code in session.go:
func (t *traceWriter) Trace(traceId []byte) {
var (
coordinator string
duration int
)
iter := t.session.control.query(`SELECT coordinator, duration
FROM system_traces.sessions
WHERE session_id = ?`, traceId)
and a couple lines below:
iter = t.session.control.query(`SELECT event_id, activity, source, source_elapsed
FROM system_traces.events
WHERE session_id = ?`, traceId)
This could be improved to make the requests token/shard-aware -- pick the right connection from the available pool of connections (as we do for user requests) based on the session_id's token.
Currently tracing queries put unnecessary strain on the cluster as they require cross-shard requests.
Lets check with the java driver if it sends tracing requests only on the control connection - if it load balances them on the regular connections - we should do the same in GoCQL (and in Scylla with shard awareness it will send requests on the correct conenction to the correct shard)
If java driver is similarly broken it should perhaps be fixed too?