kombu Has anyone looked into Cloud Pub/Sub?

trafficstars

Is anyone working on cloud pub/sub transport? It is awesome that SQS is now a first class citizen for those in the AWS world. Cloud pub/sub would do the same thing for those of us in Google who do not want to manage a rabbitmq server.
If the answer to 1 is no... if I were to develop one, would there be a chance for inclusion? Or am I better off running a contrib or a fork with my package? I think I am going to share it with the world no matter what, just not sure the right way to share it with the world. I know you guys don't want a million broken transports (and have dropped some in the past), so want to figure out the right plan to work with you guys.
Any known technical reasons why cloud pub/sub would not work? https://cloud.google.com/pubsub/ My plan was to use SQS as a model and try to port the functionality. I don't know if the specs give an immediate red flag for why it would or wouldn't work with Celery.

Thanks!!!

Mar 16 '17 13:03 listingmirror

Related issue/question from Celery issue tracker.

Apr 28 '17 05:04 georgepsarakis

@listingmirror contributions are very welcome! However, you would want to make sure that the new transport is adequately tested. A good idea would also be to see if there is some way of adding integration tests for the specific transport.

Apr 28 '17 05:04 georgepsarakis

@listingmirror I'm also interested and happy to contribute, did you create the repository?

Jun 11 '17 19:06 danigosa

I'm working on a transport for Google Pub/Sub. I just started with the memory transport and started hacking on it until something worked. I haven't found good guidelines for how to construct a new transport. I welcome collaboration. My experimental code is available here: https://github.com/safarijv/pubsub.

Jun 15 '17 19:06 brianglass

Hi, Are you considering to continue a Google Pubsub implementation ?

Regards,

Nov 13 '17 06:11 mmorel-fcms

@mmorel-fcms, we are still considering a Pubsub implementation. Google released a completely rewritten backward-incompatible client, so I am having to go back and rewrite most of the Kombu Pubsub backend that I wrote before. Due to that, work on it has somewhat stalled.

Nov 13 '17 16:11 brianglass

@mmorel-fcms we just put some more work into beefing up RabbitMQ, and it has worked out OK. A lot of the problems we were hitting seemed to be Redis issues, and those went away when we went back to RabbitMQ. 100% I would choose Cloud Pub/Sub over RabbitMQ if it worked well. I would be even willing to help test and troubleshoot. But RabbitMQ is not so bad that I would write this 100% on scratch on my own right now.. but this could definitely make it back on my radar at some point.

Nov 13 '17 17:11 listingmirror

I wish Kombu had a documentation / guide about how to create a transport. I'm happy to contribute that create Google pub/sub transport.

Dec 07 '17 11:12 afshinm

I found a possible good place to start. It seems the in-memory transport is quite basic, and you could probably swap out the implementation with GCP Pubsub more simply, since it's basically a test stub for the bare minimum functionality: https://github.com/celery/kombu/blob/master/kombu/transport/memory.py

Jan 03 '18 21:01 bbhoss

You might want to take a look at: https://github.com/GoogleCloudPlatform/psq as an option as well.

Mar 19 '18 02:03 cdnsteve

@cdnsteve psq has far far less features than Celery does.

As soon as I can spare a week, I want to get cloud pub sub going, just keep getting distracted. :(

Mar 19 '18 14:03 listingmirror

As I mentioned several months ago, I've been working on a pubsub transport for Kombu. I got distracted for a few months, but just managed to get a rough cut working. You can see it here: https://github.com/safarijv/pubsub. It still needs a lot of testing but seems to work for both raw Kombu and Celery tasks. Contributors and testers are welcome.

Mar 20 '18 22:03 brianglass

@brianglass I did see that, thanks a ton!!

Do we know the over/under on this getting approved if this is developed? I may be able to put an employee on this to flush it out / test it, if it would get merged upstream.

Mar 20 '18 23:03 brianwawok

@brianwawok I have not communicated with the Kombu team about this, so I don't know about the potential for approval. My intention is to do a little more testing and development to make sure it's solid before submitting it. I've only tested it on the 3 main ways we use it, so I imagine there are a lot of other cases that need to be fixed/tested.

Mar 20 '18 23:03 brianglass

Hi There. Any news on this Google Pub / Sub support for Celery ? Need any help ? Thanks

Apr 11 '18 13:04 n4n0r

@n4n0r, the transport in my test repo seems to be working for Celery at this point. It doesn't leverage the full functionality of pubsub yet and depends on Kombu's virtual transport adapter. I'm in the process of extending that to leverage more of the pubsub features. https://github.com/safarijv/pubsub

Apr 11 '18 22:04 brianglass

@brianglass any feeling on if there will be any shortfalls with cloud pub sub? A brief glance at the API docs told me things like countdown would not work natively, and acks late seems iffy?

Apr 14 '18 14:04 brianwawok

@brianwawok, I haven't dug deep enough to have a good sense of what all will and won't work. IMHO, the most glaring omission is lack of support for topic exchanges. Pubsub basically only supports direct exchanges. I'm playing with a pseudo topic exchange right now since that's a major aspect of how we use RabbitMQ. If that effort fails, I may have to drop this work and just deploy RMQ into the cloud.

Apr 24 '18 16:04 brianglass

@brianwawok one thing I’m a bit concerned about when looking into pubsub is late ack’s. From what I remember Google Pubsub has a maximum 10 minutes deadline until retry. This can be extended by a separate call, but it means having either a watcher thread or the user handling this in the tasks.

https://github.com/GoogleCloudPlatform/psq/issues/9 Had a good discussion of pro and cons.

But I might be an outlier considering my tasks can easily take an hour to complete, with large chunks being spent in one single function.

Would love to drop my RabbitMQ clusters for PubSub, and still continue using Celery

Jun 09 '18 15:06 bivald

I'm going to be going on a medical leave so will not be making any progress on this project in the foreseeable future. And with the announcement of Google's Cloud Memorystore, we will probably end up just using "Redis" for Celery.

Jun 11 '18 22:06 brianglass

@brianglass then I’ll possibly do the same. I wish you a speedy recovery, I hope you get well soon

Jun 12 '18 17:06 bivald

@bivald I think that the google-cloud-python library already does just that, they automatically renew the lease through a separate thread. Besides late acks (which seem to be handled in the current python library), are there any other issues with using cloud pub/sub as a broker?

I think that it should be much more robust and performant than using redis (as a broker, not result backend)...

Jun 13 '18 14:06 linar-jether

@linar-jether Interesting, they do?

From what I've gathered the rest should be fairly straight forward (I'm thinking one can look heavily on https://github.com/GoogleCloudPlatform/psq for inspiration).

Jun 13 '18 14:06 bivald

https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/leaser.py

Code is well documented, they automatically renew leases for subscriptions until the max lease time (which is configurable).

Another good thing that the python client does is request batching, meaning celery can easily submit thousands of tasks in a single roundtrip, which you currently cant do with the redis broker(although that redis pipelines as built just for that)

Jun 13 '18 14:06 linar-jether

Any news on this issue?