service-fabric-pub-sub icon indicating copy to clipboard operation
service-fabric-pub-sub copied to clipboard

Published 20 hours ago • verified publisher icon SoCreate

Is it possible to specify a custom partition scheme?

Open olivergrimes opened this issue 5 years ago • 5 comments

I could only find this method in BrokerServiceLocator that creates partition keys for connecting to the broker service:

https://github.com/SoCreate/service-fabric-pub-sub/blob/5b92715c64c5384f10fd532d4912f4cf5c1f8409/src/SoCreate.ServiceFabric.PubSub/Helpers/BrokerServiceLocator.cs#L160

Is it possible to specify a custom partition key, rather than partitioning based on the message Type? This way we could scale based on the message subject/topic as opposed to the message type.

olivergrimes avatar Nov 21 '19 17:11 olivergrimes

It doesn't currently support specifying a custom partition scheme, but that does sound like a useful feature. PR's are welcome if you want to implement it.

danadesrosiers avatar Nov 21 '19 21:11 danadesrosiers

@danadesrosiers Thanks again, if we decide to use this in our current project I'll definitely submit a PR for this feature.

olivergrimes avatar Nov 21 '19 22:11 olivergrimes

This feature is very attractive to me. I am trying to ball park the scope of work needed to implement it.

  • A new DataMember "PartitionKey" would need to be added to MessageWrapper to partition on
  • Update Broker Service to toggle between using PartitionKey and MessageType
  • Update Subscribe Attributes to support PartitionKey and MessageType
  • Other things I didn't think of?

Calidus avatar Jan 20 '21 21:01 Calidus

Can you explain what you hope to get out of this feature? The most basic enhancement I can think of would be to update BrokerServiceLocator to change private async Task<ServicePartitionKey> GetPartitionForMessageAsync(string messageTypeName) to protected virtual. That would allow the user to override it to specify which types to assign to which partition. Probably registering some kind of PartitionStrategy with BrokerServiceLocator would be a better pattern.

But that would not give you the ability to partition a single message type across multiple partitions. If you need that, I think it would be significantly more complex.

danadesrosiers avatar Jan 22 '21 23:01 danadesrosiers

After talking with some coworkers I realize I was interested in the idea of "Named Queues" rather than changing the partition scheme. Sorry for the confusion please disregard the previous comment.

Calidus avatar Jan 23 '21 14:01 Calidus