faust icon indicating copy to clipboard operation
faust copied to clipboard

Is there a way to assign the number of partitions on table dynamically relying on topic's number of partitions

Open dolshevsk opened this issue 2 years ago • 1 comments

Checklist

  • [X] I have included information about relevant versions
  • [x] I have verified that the issue persists when using the master branch of Faust.

Steps to reproduce

I want to know if there is a way to assign the number of partitions on table dynamically relying on topic's number of partitions. In my app I consume from external topics which doesn't belong to me which has different number of partitions in different environment and their number can be changed in future, but I do modify table under those topics and thus the number of partitions should be equal.

Expected behavior

I wanted to achieve it with something like that, but it doesn't work:

courier_event_topic = app.topic(
    "courier_topic",
)

courier_table = app.Table(
    "courier_table",
    default=set,
    partitions=courier_event_topic.partitions,
)

Actual behavior

right now in code above courier_event_topic.partitions will be None

Versions

  • Python version: Python 3.10.6
  • Faust version: 0.10.14
  • Operating system: Ubuntu 22
  • Kafka version: 3.4
  • RocksDB version (if applicable)

dolshevsk avatar Aug 15 '23 12:08 dolshevsk

This functionality does not currently exist in Faust. The best way I figure to go about doing this is to have a separate application track the topic metadata and propagate the number of partitions into some configuration file, which your Faust uses to determine the number of table partitions.

wbarnha avatar Aug 23 '23 21:08 wbarnha