gocql icon indicating copy to clipboard operation
gocql copied to clipboard

Make tracing requests shard aware

Open kbr- opened this issue 5 years ago • 2 comments

Currently, all tracing requests use a single connection, the control connection; this connection might or might not be "the right" connection (in terms of which shard it's connected to) for the given session_id (which is the partition key in system_traces.sessions and system_traces.events).

Relevant code in session.go:

func (t *traceWriter) Trace(traceId []byte) {
	var (
		coordinator string
		duration    int
	)
	iter := t.session.control.query(`SELECT coordinator, duration
			FROM system_traces.sessions
			WHERE session_id = ?`, traceId)

and a couple lines below:

	iter = t.session.control.query(`SELECT event_id, activity, source, source_elapsed
			FROM system_traces.events
			WHERE session_id = ?`, traceId)

This could be improved to make the requests token/shard-aware -- pick the right connection from the available pool of connections (as we do for user requests) based on the session_id's token.

Currently tracing queries put unnecessary strain on the cluster as they require cross-shard requests.

kbr- avatar Sep 17 '20 14:09 kbr-

Lets check with the java driver if it sends tracing requests only on the control connection - if it load balances them on the regular connections - we should do the same in GoCQL (and in Scylla with shard awareness it will send requests on the correct conenction to the correct shard)

slivne avatar Sep 24 '20 09:09 slivne

If java driver is similarly broken it should perhaps be fixed too?

mmatczuk avatar Sep 24 '20 09:09 mmatczuk