MQTT Integration should have a richer topic structure

[JulianFeinauer]

Currently, when using the mqtt integration one acn only subscripe all messages in one app or none. But it would be beneficial in many scenarios to get only a substream of the messages, e.g. only from one device. So it would be helpful to allow an alternative subscription topic "tree" which contains e..g the partition key or the subject of the messages. Then one could use the MQTT topic filter semantics to narrow down the messages one wants to receive.

[ctron]

That is true. But, having all messages/events in a single Kafka stream might make this tricky and provide a false impression of what is going on in the system.

With Kafka it is "all or nothing", so if you would filter for a single device, you would need to consume all messages and throw away all the other events.

However, there are a few use cases where we would like to have a similar feature.

• The firmware updater only receiving messages with a specific subject/channel, for all devices of an application, but in a way that these messages are no longer part of the dedicated message stream ← @lulf
• Messages in a "digital twin" format (e.g. identified by event type, or content-type) which get forwarded to a digital twin target/topic/stream
• Arbitrary topic filters tapping into the event stream (e.g. using the MQTT integration)

The latter one might be the most complicated. And I am not sure if the IoT connectivity layer is the right location for this. Or if it wouldn't be better to implement this in a digital twin layer. Where one could filter in something like a "virtual topic hierarchy", which maps to the digital twin space.

The other two use-cases could be implemented in some kind of pre-configured rule, which would define how messages will be delivered. Something like "if message is for firmware update, deliver to kafka topic X". Which would: 1) create a new kafka topic for the user/application 2) direct all matching messages to this topic 3) but keep them off the main event stream (which would also hide them from the MQTT integration).

[JulianFeinauer]

Hey Jens, i agree with your general analysis although one also has to distinguish between the Kafka messages that drogue has to consume and then the generated MQTT messages that have to be transferred to the client.

So i agree with your analysis but still think it could be worthwhile to have some kind of server side filtering.