Alpakka: google-cloud-pub-sub - One subscriber consuming 60% of messages

[rlupo06]

Summary:

I have an application (subscriber) that I'm scaling to 4 instances. All 4 instances are subscribers of one subscription. For some reason 1 of those instances is consuming 60% of messages on the Pub/Sub subscription (300k+ small messages on pubsub's subscription) and the other 3 are consuming about 13% each. I believe the latency of the instance consuming 60% of messages is high and I believe if I could get all 4 instances consuming a similar percentages I could increase throughput.

My application uses version 1.1.2 of the akka-stream-alpakka-google-cloud-pub-sub library which communicates to Pub/Sub through HTTP requests NOT gRPC. I've read through the release notes of the newer versions of that library and there's no mention of a change that would fix what I am seeing.

Question: Thoughts on how to get messages distributed more equally among the subscribers and any explanation on why one of the instances is consuming a much larger portion of the messages?