nestjs-kafka icon indicating copy to clipboard operation
nestjs-kafka copied to clipboard

Published 20 hours ago • verified publisher icon rob3000

Caching of latest schema ID's

Open maxgr0 opened this issue 3 years ago • 7 comments

Firstly, thanks for the work here!🚀

In KafkaAvroRequestSerializer, the latest schema id's are saved. Definitely makes sense for me - but what happens when a schema updates (and with it the correct latest id)? Do we need a mechanism for period refreshes of the latest schema id's?

maxgr0 avatar Apr 09 '21 11:04 maxgr0

Hi @maxgr0 Thanks! 😁

So the current workflow I have while using this library is to restart the service in order to pick up the new schema version. I know there has been a fair bit of re work done to the underlying library: https://github.com/kafkajs/confluent-schema-registry

By default when a new message from Kafka is received it includes the registry I'd: https://github.com/kafkajs/confluent-schema-registry/blob/master/src/SchemaRegistry.ts

Which the library gets from cache or loads from schema. We could look at similar functionality.

Have you thoughts on how you'd want to implement it?

rob3000 avatar Apr 09 '21 12:04 rob3000

deserialize is not a problem - logic here makes sense for me. Just wondering if I have a thinking error here. We're having a couple of replicas sitting behind a Kubernetes Ingress. When producer-related code updates, the replicas update successively anyways (so the schema id's are updated in any case). This raises the question if something like a TTL is really needed 🤔 What are your thoughts about this @rob3000?

I think an implementation could be easy to do, setting a variable like lastSchemaIdFetch and compare it when serialize is done. To let this be scaled up without having thousands of parallel requests to the schema registry, a second indicator e.g. isFetchingSchema could be used.

maxgr0 avatar Apr 09 '21 13:04 maxgr0

I think having something like a TTL for the serialize would make sense so that it would auto update to the latest schema without requiring intervention.

I think that implementation idea sounds great!

rob3000 avatar Apr 12 '21 01:04 rob3000

Sounds good! Do you plan to add this feature in the next time or are you busy at the moment?

maxgr0 avatar Apr 16 '21 10:04 maxgr0

Hi @maxgr0 sorry for the delay, I've pushed out this change to master, will look to get through testing this later today/tomorrow.

rob3000 avatar Apr 22 '21 00:04 rob3000

@rob3000 oh mate you're the real MVP. Please send over your PayPal, I owe you some beers!

maxgr0 avatar Apr 22 '21 10:04 maxgr0

@maxgr0 thanks! Though to be fair the guys over at KafkaJS could use the beers more than i :)

rob3000 avatar Apr 23 '21 08:04 rob3000