Particular
Topic Topology to allow bypassing the forwarding topology restrictions
This PR introduces the possibility to circumvent the limitations of the current forwarding topology by separating the publish topic from the subscribe topic.
Currently the limitations of the topology are described in our topology documentation. At first sight, it looks like we can support quite a few endpoints. In reality, this number is much smaller because the more event types you get, the more likely it is that you are running into the limit of 2000 rules since every event type SQL filter is represented as its own rule.
Relevant quotas for the discussion
As an example, in an event-driven system you may have dozens of event types for various business components to interact with each other. Just for the example, let's assume we are dealing with a system of 100 different event types. To simplify the math, let's assume every endpoint available will be on average subscribe to 25% of the events. That means every endpoint has to manage at least 25 rules, which brings us down to a scaling limit of 80 endpoints which is definitely far away from the 2000 endpoints as the documentation suggests. The more events you have and the more and endpoint is interested in subscribing to them, you get less and fewer endpoints you can start adding to your system.
Currently, when you start approaching those limits, it is only possible to go the extreme route of managing the whole topology yourself. This would mean you have to:
- Disable auto subscription on all the endpoints
- Segregate all endpoints into different topics
- Have bridges interacting between the different segments of the system
This manual management is error-prone and cumbersome. This PR attempts to solve that.
Existing systems
By default the transport uses bundle-1 for the publish topic and the subscribe topic. This makes everything compatible with existing endpoints. Once you start approaching the limits of the transport it is possible add new subscribers on a new subscription topic but still publish to bundle-1. The endpoint will then automatically add a forwarding rule.
Existing Subscriber 1 and Subscriber 2 can be left as is. Subscriber 3 has to define the hierarchy.
// Subscriber3
var transport = b.ConfigureTransport<AzureServiceBusTransport>();
transport.Topology = TopicTopology.Hierarchy("bundle-1", "bundle-2");
Should the existing system use something else than bundle-1 like the following code illustrates
// Subscriber1
var transport = b.ConfigureTransport<AzureServiceBusTransport>();
transport.TopicName = "my-custom-bundle";
// Subscriber2
var transport = b.ConfigureTransport<AzureServiceBusTransport>();
transport.TopicName = "my-custom-bundle";
Subscriber 3 would have to define the hierarchy like
// Subscriber3
var transport = b.ConfigureTransport<AzureServiceBusTransport>();
transport.Topology = TopicTopology.Hierarchy("my-custom-bundle", "my-custom-sub-bundle");
New systems
Can choose freely how the want to segregate things into different "bundle topics" or start with the defaults and then later switch as described above.
Caveat
- The maximum number of forwarding hops is 4. The "sub" topics increase the hops by one but it still falls into the limit.
- In theory things could be nested further as long as it fits into the maximum number of 4 hops.
- For standard namespaces this will increase slightly the costs because forwarding operations will also be charged. Premium namespace users are billed per messaging unit (which is CPU and memory) and are not affected.
Missing part
- [x] Making the CLI tool support this
