Extract underylying technology stack into a clearly Fedora independent library?

[ncoghlan]

(Moving an in-person conversation with @ralphbean to a linkable location)

Currently, fedmsg is still documented at fedmsg.com and here on GitHub as either Fedora Messaging or the Fedora Infrastructure Message Bus. However, the underlying technology stack provides a brokerless JSON-based "to whom it may concern" event publishing system that enables the development of web services that are useful as standalone applications, but nevertheless support robust programmatic monitoring and event-based triggers that span multiple services.

One of my personal points of concern is that I'd love to see more talks about this system at upstream Python conferences (and subsequent increased adoption beyond Fedora), but as things currently stand, most review committees would turn down related talk proposals as off-topic, as there's no clear distinction between the underlying technology and Fedora's deployment of that technology.

[ralphbean]

+1

/cc @abompard, @sayanchowdhury

[thequbit]

:+1: ... would love to see this pulled out as a stand alone set of tools :).

[abompard]

That's totally the kind of work I'd be interested in doing :-) But it may be a bit early just yet wrt my understanding of Fedmsg. I'm working on that though.

[pypingou]

Is the issue documentation (fedmsg == federating message bus != fedora message bus) or are they actual changes needed in the code?

[ralphbean]

I think the idea is to take almost all the code in fedmsg and move it to a new module with a new independent name like pingoulib (or something like that).

And then we'd take fedmsg/__init__.py and rewrite it to just use pingoulib, like:

import functools
import pingoulib

@functools.wraps(pingoulib.publish)
def publish(*args, **kwargs):
    return pingoulib.publish(*args, **kwargs)

@functools.wraps(pingoulib.tail_messages)
def tail_messages(*args, **kwargs):
    for items in pingoulib.tail_messages(*args, **kwargs):
        yield items

So calling fedmsg.publish(..) would do just pass execution onto our new independent lib. We wouldn't have to update any of our apps, ideally.

[pypingou]

So it's sort of renaming fedmsg to a something to drop the fed part?

[skrzepto]

I think the idea is to take almost all the code in fedmsg and move it to a new module with a new independent name

I like the idea of moving to a new module but how about changing the name to what pypingou mentioned

fedmsg== federating message bus

[sayanchowdhury]

+1 to @pypingou said. I have already seen @ralphbean mentioning fedmsg as federated message bus. We can remove the fedora references from fedmsg documentation and update https://fedora-fedmsg.readthedocs.io/ with how fedora uses fedmsg and fedmsg-apps references ?

[ralphbean]

@ncoghlan would that be sufficient?

[ncoghlan]

I think if the docs used the term "Federated Message Bus" consistently (perhaps with a history section explaining the origins as "Fedora Messaging" and "Fedora Infrastructure Message Bus") that would likely be sufficient from a Python upstream point of view - the issue I see at the moment is that a talk reviewer may see "fedmsg" in a proposal, look it up on Google, and think they're going to get a talk specifically about Fedora Infrastructure rather than a general purpose library and reject it as off-topic for a Python conference.

Referencing fedora-fedmsg would definitely help with that, since it means the distinction between fedmsg-the-library and fedmsg-the-Fedora-Infrastructure-data-feed already exists.

[ncoghlan]

Since I don't think I've written it down anywhere else, I'll also add a "Why fedmsg is cool" comment that may be useful in more clearly separating the fedora-fedmsg docs from the fedmsg ones: unlike many structured event publishing systems, fedmsg lets you publish structured events from your service without making any assumptions about who's listening - you only need suitable configuration settings in order to subscribe, not to publish.

This then ties in nicely with platforms like Kubernetes and Docker Compose, which have service discovery mechanisms built in - fedmsg makes it trivial to use different ways of obtaining the data feed host and port information in different environments (falling back on DNS as a lowest common denominator option), rather than assuming a grand-unified message bus that all components of the system are necessarily aware of.

fedmsg also pairs up nicely with using HTTPS+JSON for synchronous request/response operations, rather than trying to use the message bus for both synchronous requests and asynchronous notifications.