bitbouncer
Would KSPP be useful for me?
Hi all. While I know this isn’t an issue I don’t know where else to ask this.
I am currently developing a Kafka application to stream live stock trading prices in real time. Most of the project is complete but I am having a hard time understanding how to join Kafka streams together.
I have one topic in the application, where each partition in that topic is assigned a stock ticker.
Without going into detail why, what I need to do is to be able to join 2 or 3 Kafka partitions together so I can do real time mathematical computations based off the message values of each partition stream.
I tried looking at some of the examples but I am bit confused if what I am trying to do is possible with this library.
Any help would be appreciated. Thanks.
well it depends. To do joins you normally do one of two things. You either join a partition of a stream with a single partition topic (hopefully small) or you join the same partition of two topics. What you are suggesting is two join 1 partition with 2-3 others. conceptually this would be like first to merge the 2-3 partitions to one and then finally to join this with the first. This sounds like an explosion if you have 10k partitions (like us nasdaq). And with kspp you would have 10k processes. This said - maybe you can design the partitioning of the first topic better than per ticker. Finally nasdaq sells a kafka stream where you get the full depth feed in one partition - this makes me believe you have more partitions than you need.
