kafka-elasticsearch-injector icon indicating copy to clipboard operation
kafka-elasticsearch-injector copied to clipboard

SchemaRegistry cache my fail in concurrent miss scenario

Open xmlking opened this issue 3 years ago • 1 comments

I this following code may not work in concurrent miss scenario https://github.com/inloco/kafka-elasticsearch-injector/blob/master/src/kafka/encode_decode.go#L115-L142

the solution described here with ChanLocker pattern might fix it. https://lakefs.io/2020/09/23/in-process-caching-in-go-scaling-lakefs-to-100k-requests-second/

xmlking avatar Sep 25 '20 21:09 xmlking

Hi @xmlking! Thanks for raising this issue.

The current code does load the schema multiple times in a concurrent miss scenario, but the performance does not suffer that much, as the number of schemas is not large (for most cases) and the loading time is not prohibitive.

The ChanLocker pattern looks like a great idea, though. PRs are welcome!

racevedoo avatar Sep 28 '20 11:09 racevedoo