super-rag icon indicating copy to clipboard operation
super-rag copied to clipboard

Published 20 hours ago • verified publisher icon superagent-ai

Add Kafka

Open elisalimli opened this issue 11 months ago • 4 comments

What does this PR do?

image

Fixes #39 #53

Steps to test

  1. Run docker-compose up -d to spin up kafka, kafka-ui and redis
  2. Start python server
  3. Run cd ./service/kafka/ & python3 ./consume.py

elisalimli avatar Mar 11 '24 06:03 elisalimli

Hey @elisalimli, I am just taking a look at ray which can simplify the integration

Also worth reading distributed kafka consumers with ray

teocns avatar Mar 11 '24 21:03 teocns

I am testing this in a Kubernetes cluster and provide feedback along the way;

teocns avatar Mar 11 '24 21:03 teocns

Hey @elisalimli, I am just taking a look at ray which can simplify the integration

Also worth reading distributed kafka consumers with ray

@teocns I haven't looked at it yet. What would be the benefit of using Ray instead of just Kafka?

elisalimli avatar Mar 12 '24 04:03 elisalimli

@homanp we can deploy this

elisalimli avatar Mar 16 '24 07:03 elisalimli