hyades icon indicating copy to clipboard operation
hyades copied to clipboard

Investigate usage of Confluent's parallel consumer

Open nscuro opened this issue 2 years ago • 1 comments

Confluent has published a wrapper around Kafka's Consumer/Producer API that enables parallel processing of messages, and supports batching and retries: https://github.com/confluentinc/parallel-consumer

parallel-consumer can be used alongside Kafka Streams in the same application. It can be hooked into vert.x (which ships with Quarkus) to perform I/O operations.

Based on what I've seen so far, parallel-consumer may be a good fit for vulnerability analysis and repo meta analysis.

When it comes to the supported ordering guarantees, I think key-based ordering should suffice. These ordering guarantees are something that Quarkus' reactive messaging implementation is currently missing, see https://github.com/DependencyTrack/hyades/issues/215#issuecomment-1377347537

nscuro avatar Feb 27 '23 10:02 nscuro

Relates to #215

nscuro avatar Feb 27 '23 10:02 nscuro

parallel-consumer was adopted in API server and notification-publisher.

nscuro avatar Jun 05 '24 10:06 nscuro