googleapis
Google Pubsub Publisher (latency/throughput config not exposed)
We are using google pubsub for a banking/transaction system and are seeing messages sometimes take as long as 17 seconds to publish using the API that is available here.
I see in other languages we have batchsize, data size and other things exposed to decrease latency, but I haven't been able to use/locate/configure these settings in elixir.
We would like to keep gPubsub, but if we can't get a handle on these performance issues in elixir specifically with publishing google pubsub messages soon we will have to find a different solution for our message queues.
Any help or guidance would be greatly appreciated.
of course, sorry to mention but time is of the most importance.
It's likely that the difference is the use of the gRPC pubsub endpoints vs HTTP/REST pubsub endpoints. Some of the more common languages have high-performance handwritten pubsub clients targeting gRPC, which is how they are able to support very high throughput and low latency. We haven't had the resources to write one for Elixir, so we're limited to an autogenerated HTTP/REST client which doesn't have as many features.
My advice at this point, if you are encountering unacceptable latency issues with the existing REST client that do not seem to appear in other languages, is to look for a different solution.
The cjab/weddell project is a good alternative: https://github.com/cjab/weddell
