Non Token Aware queries for counters

[dahankzter]

The Non Token Aware Queries graph doesn't seem to be accurate for counters. We have tested both the Go driver (https://github.com/scylladb/gocql/issues/32) and cassandra-stress and both of them shows a large number in this graph but none when switching to integer fields or text for example.

cassandra-stress run:

cassandra-stress counter_write no-warmup cl=QUORUM duration=30s -schema 'replication(factor=2) compaction(strategy=SizeTieredCompactionStrategy)' keyspace=counter -port jmx=7100 -mode cql3 native -rate threads=1

This is both on multi node setups and even single node clusters.

[dahankzter]

[jul-stas]

For future readers: it also might be worth noting that:

• we used shard-aware drivers
• scylla_storage_proxy_replica_cross_shard_ops metrics don't indicate any inter-shard operations
• the behavior is observable even for Scylla ran with --smp 1

[slivne]

@amnonh ping ...

[amnonh]

@slivne you volunteered to have a look, let me know if you can't

[gcarmin]

@slivne @amnonh any update on this one?

[gcarmin]

@slivne @amnonh any update on this one?

[slivne]

this is the source of the issue: https://github.com/scylladb/scylla/issues/4337

I'll need to followup with gleb on this.

[amnonh]

@slivne any updates?

[amnonh]

@slivne ping

[amnonh]

@slivne please comment to know if we want to close this

[gcarmin]

@vladzcloudius FYI

[slivne]

The counter today indeed is not updated and token-awareness in the dashboards is incorrectly displayed

we will either

• add counters for local/remote updates of counters
• remove the number of updates from counters for dashboards to make sure the input on token awarness is correct

[mrsinham]

@amnonh Vlad told me to ping you on this issue!

[amnonh]

@slivne we need to link to a scylla issue

[slivne]

@jul-stas is looking at this https://github.com/scylladb/scylla/issues/4337

@amnonh you can also try and have a temp solution by removing counter updates for now - it will not be as good but will be a stop the gap till scylla releases will havea solution

[amnonh]

There is a workaround in Scylla Monitoring 3.6.2

[gcarmin]

@amnonh what is the workaround?

[amnonh]

when counters or LWT in use, it will not show non-token aware