Porting from fedmsg to fedora-messaging is not smooth

[TomasTomecek]

I am trying to migrate our project from fedmsg to fedora-messaging.

Right now I'm stuck on the config file:

• while looking at the one provided by fedora-messaging rpm, it feels like it's just a placeholder and not meant to be used for production
• there is a ton of values prefixed with my_ while some seem to have real values
• amqp_url is not filled out -- how do I figure out the prod and stage instances?
• when doing api.consume by default, it connects to localhost -- not very useful

So, what are the best practices when it comes to configuration?

I did also try the fedora-messaing executable:

$ fedora-messaging consume --routing-key org.fedoraproject.prod.github.pull_request.*
Error: You must define all three of exchange, queue_name and routing_key, or none of them to use the configuration

Ehm, the --help says there are default values. Okay then:

$ fedora-messaging consume --routing-key org.fedoraproject.prod.github.pull_request.* --exchange amq.topic --queue-name asdqwe123123
Error: Failed to import the callback module (No module named 'fedora_messaging.examples')

:(

[jeremycline]

On 1/7/19 11:32 AM, Tomas Tomecek wrote:

I am trying to migrate our project from fedmsg to fedora-messaging.

Thanks for filing a bug, you're the first person to try this as far as I know.

Right now I'm stuck on the config file:

• while looking at the one provided by fedora-messaging rpm, it feels like it's just a placeholder and not meant to be used for production

That is definitely the case. It was designed to work "out of the box" with a local message broker for quick and easy development environments. Did you find the configuration documentation[0]? If you didn't, we should definitely make that easier to find, and if you did and it didn't help we should fix the docs.

[0] https://fedora-messaging.readthedocs.io/en/stable/configuration.html

• there is a ton of values prefixed with my_ while some seem to have real values

Indeed. Most settings are application-dependent (path to client cert, queue name and routing keys, etc). Perhaps inline comments in addition to the documentation page is required to make it easy to dive right in.

• amqp_url is not filled out -- how do I figure out the prod and stage instances?

The hosts are in the Fedora Ansible repository. They aren't in the default config for a few reasons, the largest being they don't accept unauthenticated connections, nor do they allow connections from the Internet so you wouldn't be able to use them.

I'm pretty sure this isn't actually written down anywhere yet, though, so I will make sure that gets done.

• when doing api.consume by default, it connects to localhost -- not very useful

So, what are the best practices when it comes to configuration?

I did also try the fedora-messaing executable:

$ fedora-messaging consume --routing-key org.fedoraproject.prod.github.pull_request.*
Error: You must define all three of exchange, queue_name and routing_key, or none of them to use the configuration

Ehm, the --help says there are default values. Okay then:

It does seem like that should work, I don't recall why it doesn't. I'll look into this as well and see if there was any reason for it.

$ fedora-messaging consume --routing-key org.fedoraproject.prod.github.pull_request.* --exchange amq.topic --queue-name asdqwe123123
Error: Failed to import the callback module (No module named 'fedora_messaging.examples')

This is the default callback that just prints to stdout, but the old example config had the wrong Python path. It got fixed in in commit a96b02e1f225 and once I see about the other problems in this issue I'll tag a release and get it out ASAP.

:(

Thanks for trying this out. I'm sorry it has been so rough. I want it to be very smooth, so I'm going to make sure these issues get fixed.

• Jeremy

[TomasTomecek]

That is definitely the case. It was designed to work "out of the box" with a local message broker for quick and easy development environments.

Oh, so that's the mismatch. My intent was to utilize the production environment, not to play with a local broker.

The hosts are in the Fedora Ansible repository. They aren't in the default config for a few reasons, the largest being they don't accept unauthenticated connections, nor do they allow connections from the Internet so you wouldn't be able to use them.

What are the plans then? When can my application start using production deployment of fedora-messaging instead of fedmsg?

The reason I'm asking is that I need write certain functionality (within next 2 weeks) and I can either do it with fedmsg (which is deprecated, right?) or fedora-messaging.

Jeremy, thanks for your thorough response.

[jeremycline]

On 1/8/19 4:03 AM, Tomas Tomecek wrote:

That is definitely the case. It was designed to work "out of the box" with a local message broker for quick and easy development environments.

Oh, so that's the mismatch. My intent was to utilize the production environment, not to play with a local broker.

The hosts are in the Fedora Ansible repository. They aren't in the default config for a few reasons, the largest being they don't accept unauthenticated connections, nor do they allow connections from the Internet so you wouldn't be able to use them.

What are the plans then? When can my application start using production deployment of fedora-messaging instead of fedmsg?

The current plan is to not allow unauthenticated client connections or open it up to the Internet at large.

The reason I'm asking is that I need write certain functionality (within next 2 weeks) and I can either do it with fedmsg (which is deprecated, right?) or fedora-messaging.

Can you tell me a bit more about what you're making and its requirements? I gather it's not going to be run by Fedora infrastructure?

If it's not running inside Fedora infrastructure, unfortunately fedmsg is probably your easiest route forward with that kind of timeline.

I assumed most people making use of this would be running in Fedora infrastructure. I've got vague plans on how to make it easy for external users to enjoy the same schema validation and APIs, but nothing that would be implemented and deployed in 2 weeks, I'm afraid.

[yaneti]

Just wanted to chime in as a current fedmsg user outside the Fedora infrastructure.

I want the simplest kind of tail-message for koji build status changes, used for downloading the recent builds for my packages of interest

I've been using

config = fedmsg.config.load_config()
for name, endpoint, topic, msg in fedmsg.tail_messages(topic='org.fedoraproject.prod.buildsys.build.state.change', mute=True, **config):
....

until 2 days ago when it seem that the public fedmsg endpoint stopped receiving messages of this topic. Or something else around fedmsg changed without me noticing

[jeremycline]

It's a known issue (https://pagure.io/fedora-infrastructure/issue/7493) with infra, seemingly related to the upgraded proxies. It is a convenient opportunity to see how many folks outside infra are consuming the messages, though!

We would definitely make this as smooth a transition as possible with plenty of time to migrate from the old to the new approach.

[jeremycline]

Hi @TomasTomecek,

The Fedora message broker is now available outside of Fedora infrastructure and the RPM comes with a configuration file aimed at connecting to it. If it's not too much trouble, can you read through https://fedora-messaging.readthedocs.io/en/latest/fedora-broker.html, try out fedora-messaging-1.6.0 (in updates-testing for Fedora 29 and 30), and let me know if anything seems confusing or difficult? I'd really appreciate it; I want this to be a transition that people actually want to do.

[TomasTomecek]

@jeremycline thanks for the update! Unfortunately, I won't have a spare cycle in coming weeks to play with this :/

[dustymabe]

Since this ticket touches on friction related to moving from fedmsg followers might find the RFE I just made in https://github.com/fedora-infra/fedora-messaging/issues/173 interesting.