fleet-telemetry Feature Request: Configurable Partition Strategy

I'd like to be able to guarantee order per vehicle by setting the partitioning strategy to hash instead of the default strategy (least bytes), which would guarantee that all messages for the same vehicle go to the same partition. This seems relatively easy to do as Kafka has a built in balancer option that hashes by the message key, which appears to be already set to the vehicle vin.

Feb 06 '25 18:02 efriesner

Reading the code, I'm surprised this isn't happening already. Happy to accept a PR.

Feb 14 '25 01:02 patrickdemers6

Maybe I'm misunderstanding you, but: can you do this already? The README says you can set librdkafka settings in the kafka section of the config, and we've been setting "partitioner": "murmur2".

Feb 19 '25 20:02 elffjs

You are right, will try this out. Thanks!

Feb 25 '25 19:02 efriesner

@patrickdemers6 it does look like partitioning by VIN is already happening without needing to change the kafka settings. Where is that getting set in the existing implementation?

Mar 05 '25 23:03 efriesner

@patrickdemers6 it does look like partitioning by VIN is already happening without needing to change the kafka settings. Where is that getting set in the existing implementation?

see all supported kafka settings from https://github.com/teslamotors/fleet-telemetry/blob/main/config/config.go#L61 the default partitioner is consistent_random which partitions based on key. Key here is VIN

Mar 06 '25 00:03 agbpatro