cassandra icon indicating copy to clipboard operation
cassandra copied to clipboard

Track internode msg bytes and counts

Open aymkhalil opened this issue 1 year ago • 1 comments

Draft PR on tracking internode message bytes and counts.

More generic implementation (i.e. leveraging InboundMessageHandlers like InternodeInboundMetrics have the following complications:

  • The work if we want to track internode messages globally. However per keyspace and table dimensions would require access the the deserialzed message and inverting the payload at runtime.
  • The life cycle of RequestSensors today (from sensor registration to synchronizing the sensors registry) begins on VerbHandler#doVerb. Trying to register and synchronise outside the verb handler boundaries to increment internode sensors would require major changes (probably where the verbs are dispatched in InboundSink)

aymkhalil avatar Apr 12 '24 19:04 aymkhalil

  • Update internode sensors tracking to happen at the outboundSink level
  • Encoded KEYSPACE header to simplify encoding internode sensors in the message header. Please note that it is not possible to link the message to a keyspace in the Mutation case because the payload is empty. Even for reads, the message doesn't naturally hold keyspace info in the LocalDataResponse and even if it does, extracting will require inspecting the payload at runtime. I found the header approach to be simpler.
  • Internode sensors are now tracking at the keyspace level only, consequently, there are only tracking in the SensorsRegistry
  • Added SensorsInternodeMessageTest unit tests

https://github.com/datastax/cassandra/pull/1081/commits/919d1cfa6b8505dbab01092dd47d475183c73db3

aymkhalil avatar Apr 17 '24 23:04 aymkhalil

Fixed PR diff https://github.com/datastax/cassandra/pull/1081/commits/cc52860bedddc46d7ea56aafaa37ffcc97059826

aymkhalil avatar May 13 '24 19:05 aymkhalil

@sbtourist PR is ready for second round of review. PTAL.

aymkhalil avatar May 13 '24 23:05 aymkhalil