wire-server icon indicating copy to clipboard operation
wire-server copied to clipboard

Implement XMPP federation

Open stevenroose opened this issue 6 years ago • 34 comments

I can't remember where or from who, but a few months ago I picked up a message that Wire wanted to implement federation between Wire servers.

I just want to give you guys a giant heads-up for the idea and meanwhile suggest using the XMPP protocol specifically for doing so. While it might seem a bit harder at first than just specifying your own custom protocol, XMPP is a very solid and extensible protocol with 20 years of R&D invested into it.

It's possible to cherry-pick extensions that make sense for Wire and there is a lot of tooling available for speaking XMPP ([1], [2], [3]) in Haskell.

As an added benefit, federating over XMPP allows all Wire users to chat with existing users on Jabber and to join existing public Jabber chat rooms. It would be an awesome step in the direction of an open federated chat network instead of a group of silo'd private solutions.

I'm sure if you have questions or doubts regarding this, there would be plenty of people with thorough knowledge on XMPP federation that are willing to help you out. ([1], [2], [3])

EDIT: some more thoughts can be found in this comment: https://github.com/wireapp/wire-server/issues/631#issuecomment-502490687

stevenroose avatar Feb 14 '19 16:02 stevenroose

If I understand correctly, this involves basically ditching our own existing E2E protocol (https://github.com/wireapp/proteus), moving to something like OMEMO, and implementing OMEMO support in all our clients.

Then, whenever we want to improve our protocol, we need to publish a XEP so that other Jabber clients can implement this XEP (if they want to be able to talk with Wire users). Alternatively we might have our own protocol that we can iterate on, and keep OMEMO as a fallback for non-Wire users.

This sounds rather significantly more involved than "might seem a bit harder at first".

Or am I missing something?

neongreen avatar Feb 14 '19 17:02 neongreen

Also: I'm not familiar with XMPP federation, so I'm curious to know whether adopting XMPP will make federation easier for us even if we decide that we still need to run our custom protocol on top of it.

neongreen avatar Feb 14 '19 17:02 neongreen

It depends on how much interoperability with the existing XMPP world you want. It's perfectly reasonable to have your own private extensions. OMEMO is an extension, there are other E2EE extensions. In any case, being able to draw from 20 years of experience can't hurt, can it?

Zash avatar Feb 14 '19 17:02 Zash

whenever we want to improve our protocol, we need to publish a XEP [...]. This sounds rather significantly more involved than "might seem a bit harder at first".

Yes: Standardization, negotiation with other players, debugging interop issues etc. certainly involves additional effort. I think interop with other systems is clearly in the interest of customers (and their external contacts), but obviously it's less clear from the vendor's point of view. Capitalism sucks :smile:

However, advantages might include:

  • The "20 years of experience" point that has been made above. You might avoid some pitfalls by not reinventing the federation wheel from scratch.
  • The external feedback you get when publishing spec proposals/updates.
  • Additional reputation in some parts of the more 'geeky' audience, which might help with marketing.
  • As your competitors don't seem to be convinced of this model, it might be another unique selling point :sweat_smile:

weiss avatar Feb 14 '19 18:02 weiss

Oh, just in case: I wasn't implying "this is not trivial so why are you even proposing this". Far from it.

I will, however, admit that I'm sceptical about standardization not being an incredibly time- and money-consuming process. That's why "a bit harder" felt out of place in the original post.

Please feel free to use this issue to post links / papers / reading material of any kind that might help us make a more informed decision (when we actually get to discussing federation internally).

On Thu, Feb 14, 2019, 19:18 Holger Weiß <[email protected] wrote:

whenever we want to improve our protocol, we need to publish a XEP [...]. This sounds rather significantly more involved than "might seem a bit harder at first".

Yes: Standardization, negotiation with other players, debugging interop issues etc. certainly involves additional effort. I think interop with other systems is clearly in the interest of customers (and their external contacts), but obviously it's less clear from the vendor's point of view. Capitalism sucks 😄

However, advantages might include:

  • The "20 years of experience" point that has been made above. You might avoid some pitfalls by not reinventing the federation wheel from scratch.
  • The external feedback you get when publishing spec proposals/updates.
  • Additional reputation in some parts of the more 'geeky' audience, which might help with marketing.
  • As your competitors don't seem to be convinced of this model, it might be another unique selling point 😅

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/wireapp/wire-server/issues/631#issuecomment-463733852, or mute the thread https://github.com/notifications/unsubscribe-auth/ABc-aqzTRxpdQ4HoK4ZyhcNGVVzBDRyLks5vNahpgaJpZM4a8BgZ .

neongreen avatar Feb 14 '19 18:02 neongreen

If I understand correctly, this involves basically ditching our own existing E2E protocol (https://github.com/wireapp/proteus), moving to something like OMEMO, and implementing OMEMO support in all our clients.

Not necessarily. You can decide both ways: either (1) you keep using your own E2EE scheme and implement it as a proprietary extension (when using a good XMPP library, it should be trivial to wrap that scheme in an extension); this has the downside that users will only be able to use this E2EE with other Wire servers that know this module. Or (2) you decide that it's valuable for Wire users to be able to chat E2EE with non-Wire users as well and implement OMEMO. It's of course also possible to have both so that users with older Wire clients can still use the Wire E2EE with other Wire users, while users that updated their client can use OMEMO. It's even possible to use both depending on what your contact supports.

(Related: Like Proteus, OMEMO is also based on the Signal double ratchet E2EE protocol (Axolotl), it might be possible to make them compatible even.)

Then, whenever we want to improve our protocol, we need to publish a XEP so that other Jabber clients can implement this XEP (if they want to be able to talk with Wire users). Alternatively we might have our own protocol that we can iterate on, and keep OMEMO as a fallback for non-Wire users.

This very much depends on how much you want to give back to the community. Like I mentioned above, it's certainly possible to just craft your own extensions without having to share them. However all functionality that you want to be adopted by the wider XMPP community so that it is compatible with other servers will have to go through the XEP standards process. Most likely there will already be a XEP for most features you'd want to implement, though.

Also, note that I'm only talking about the server-to-server XMPP protocol. A lot of the XMPP complexity is in the client-to-server protocol, which you don't have to implement. It's perfectly reasonable to decide that Wire users should only use the official Wire app and it's OK to have your own c2s protocol between the Wire server and client.

stevenroose avatar Feb 15 '19 10:02 stevenroose

Also, note that I'm only talking about the server-to-server XMPP protocol. A lot of the XMPP complexity is in the client-to-server protocol, which you don't have to implement. It's perfectly reasonable to decide that Wire users should only use the official Wire app and it's OK to have your own c2s protocol between the Wire server and client.

Oh, that's interesting to know. I didn't realise this.

neongreen avatar Feb 15 '19 10:02 neongreen

I honestly do not think XMPP is a good idea. XMPP, in my opinion, is a badly designed protocol that doesn't enforce even the most basic XEP guidelines for implementing your own server, this means that there's a ridiculous amount of fragmentation between servers and clients. In most cases, users are completely blind and unable to check if a certain server supports certain features (such as push notifications, which are absolutely essential for a good mobile experience, and this is really not up for debate).

Some server software implement some XEPs, while other client software implement other XEPs. Like @neongreen mentioned, E2E encryption is also not part of the protocol, as it is implemented client-side through OMEMO (then again that depends on specific clients supporting OMEMO, as opposed to OTR and PGP which are also encryption mechanisms for XMPP). This is similar to e-mail, where you have to add a layer of encryption through PGP. And we all know what the problem is: The lowest common denominator is cleartext, and I find this completely unacceptable in a communication software for 2019. There really is no excuse for a protocol not to use full E2E encryption nowadays.

XMPP, however, is very useful for systems such as in-game chats (let's say World of Warcraft) or for M2M communication. Nintendo recently said they use XMPP for push notification on their Switch console. That's a very good idea, because XMPP being so modular means developers can adapt it to their use case, implementing the XEPs they want and leaving out the ones they don't consider important.

However, as a means of personal communication I find XMPP as part of the past. We should move on.

L1Cafe avatar Jun 16 '19 10:06 L1Cafe

I disagree with many details; e.g., while you consider it "completely unacceptable" to not enforce E2EE "in 2019", for me it's unacceptable to have my communication rely on closed services that allow me to talk to their own customers only, in 2019. As E2EE is especially hard to enforce on open networks, there's a trade-off, so I don't think there's a "right or wrong" here. The difference is that E2EE can be implemented on an open network, while there's no way for us to interop with closed silos.

That said, I do agree with your general stance: Fragmentation can obviously result in annoying UX, and XMPP suffers from this. The main problem here is missing manpower (i.e., money) being invested into the ecosystem, hence only some implementations are currently up to par with modern (mobile) features, so users must pick the right combinations, which is not something they want to be bothered with. However, I disagree the protocol's modularity is the culprit here. Quite the opposite, the extensibility is essential for moving the protocol forward, i.e. for adapting to new and different use cases (such as today's mobile IM requirements) without breaking interop with the existing ecosystem. Due to feature negotiation, it's of course easy to check whether a certain functionality (such as push notifications) is available. (No, users shouldn't have to check this, the apps they install should just offer sane services by default.)

Anyway, this discussion seems quite off-topic to me, as this issue is not about talking anyone into another solution or comparing Wire to XMPP. Your preferences are different from mine, and that's obviously fine. It's just about whether it would be nice to open an existing solution for communication with others, so that the two of us could chat with each other (just like we could call us by phone). If enforcing E2EE is considered essential by one of the vendors involved, just don't offer a knob to disable it in your client. Or wouldn't that solve your problem?

weiss avatar Jun 16 '19 11:06 weiss

I just want to also bring https://matrix.org/docs/spec/ to attention, it's in my opinion better than XMPP.

swedneck avatar Jun 16 '19 12:06 swedneck

it's in my opinion better than XMPP.

How?

akuckartz avatar Jun 16 '19 12:06 akuckartz

If enforcing E2EE is considered essential by one of the vendors involved, just don't offer a knob to disable it in your client. Or wouldn't that solve your problem?

Yes and no. The basic XMPP specification requires clients to be able to talk in cleartext. If you forced OMEMO in the Wire app to be able to talk to other XMPP clients, you wouldn't actually be adhering to the XMPP specification, would you?

This is similar to, for example, having an "FTP client" that only accepts FTPES. Or a web browser that cannot browse cleartext HTTP websites: It's not a web browser, it's an HTTPS browser, if at all.

L1Cafe avatar Jun 16 '19 13:06 L1Cafe

If enforcing E2EE is considered essential by one of the vendors involved, just don't offer a knob to disable it in your client. Or wouldn't that solve your problem?

What about incoming messages without E2EE enabled, or messages to XMPP users who’s clients don’t support E2EE? I’m not really familiar with XMPP or how it would handle those scenarios, but it doesn’t seem like the UX Wire users are looking for: seamless always-on E2EE. And I’d consider it unacceptable for E2EE to be disabled by either client in either scenario.

I’d imagine the Wire server federation being planned will work a lot more seamlessly than trying to force XMPP to act the way Wire expects it to. Otherwise why put all this work into Wire’s server and encryption in the first place?

jonaharagon avatar Jun 16 '19 13:06 jonaharagon

The basic XMPP specification requires clients to be able to talk in cleartext.

Can you point me to the paragraph in the XMPP spec were it says clients MUST be able to send and receive plain text messages?

iNPUTmice avatar Jun 16 '19 13:06 iNPUTmice

Can you point me to the paragraph in the XMPP spec were it says clients MUST be able to send and receive plain text messages?

You understand that the X in XMPP means "extensible" and that XEPs are not part of the core protocol? It gets even worse if we stick to the specifications because OMEMO is deemed experimental: https://xmpp.org/extensions/xep-0384.html

The other alternative is OpenPGP, which is also experimental: https://xmpp.org/extensions/xep-0373.html

The situation with server to client and client to client encryption on TLS and SSL is not any better https://xmpp.org/extensions/xep-0250.html https://xmpp.org/extensions/xep-0035.html

Do you really think such a fragmented system should be the basis for Wire? I really don't. We can do better than XMPP.

L1Cafe avatar Jun 16 '19 13:06 L1Cafe

@L1Cafe you said the ›The basic XMPP specification requires clients to be able to talk in cleartext‹ and I was simply asking you to back this statement up with a quote from the spec.

iNPUTmice avatar Jun 16 '19 13:06 iNPUTmice

Correction: TLS encryption is part of XMPP, but that only concerns client to server communication. We still have no standard way of client to client E2E encryption.

L1Cafe avatar Jun 16 '19 13:06 L1Cafe

TLS encryption is part of XMPP, but that only concerns client to server communication.

The core XMPP RFC specifies TLS for server-to-server connections just fine (for ages), and there's no problem in practice at all. I don't think such half-informed comments are useful.

But yes, standardization requires time and efforts, and yes, you don't have this issue when just hacking together a closed solution. I still believe discussion of the pros and cons of these models is off-topic for this issue.

weiss avatar Jun 16 '19 14:06 weiss

I don't think such half-informed comments are useful.

I don't think such passive-aggressive comments are useful either. I already corrected myself.

you don't have this issue when just hacking together a closed solution

You are misrepresenting my arguments. I am not suggesting Wire rolls their own closed solution. Wire is open source, both client and server. I see no reason why you would infer I want Wire to use a closed source federated protocol. Actually, I'm implicitly suggesting the opposite: We can do better than XMPP if we roll out a newer, improved protocol. Someone suggested Matrix already, with which I disagree because I have my reservations on the competence of the core developers, but Matrix has already solved many issues of XMPP, like this precise E2E problem I am trying to highlight.

I still believe discussion of the pros and cons of these models is off-topic for this issue.

As far as I can see, we are discussing XMPP federation on Wire. I am offering my opinions on why this is a bad idea. You offered some advantages of sticking to XMPP, as well, https://github.com/wireapp/wire-server/issues/631#issuecomment-463733852.

L1Cafe avatar Jun 16 '19 14:06 L1Cafe

I honestly do not think XMPP is a good idea. XMPP, in my opinion, is a badly designed protocol that doesn't enforce even the most basic XEP guidelines for implementing your own server, this means that there's a ridiculous amount of fragmentation between servers and clients. In most cases, users are completely blind and unable to check if a certain server supports certain features (such as push notifications, which are absolutely essential for a good mobile experience, and this is really not up for debate).

This is false. The XMPP protocol has ample way to discover which features and services are supported by your server. That's crucial when a protocol is extensible and doesn't force a large set of minimal supported features upon implementers.

If you forced OMEMO in the Wire app to be able to talk to other XMPP clients, you wouldn't actually be adhering to the XMPP specification, would you?

This argument is wrong. Perfectly possible to show a message "this contact's server does not support encrypted chatting" when a Wire user adds a Jabber user that doesn't support OMEMO. It's up to Wire to decide whether it wants to allow the user to decide to communicate in cleartext or not allow communication. Both seem fine decisions IMHO.

What about incoming messages without E2EE enabled, or messages to XMPP users who’s clients don’t support E2EE? I’m not really familiar with XMPP or how it would handle those scenarios, but it doesn’t seem like the UX Wire users are looking for: seamless always-on E2EE. And I’d consider it unacceptable for E2EE to be disabled by either client in either scenario.

I’d imagine the Wire server federation being planned will work a lot more seamlessly than trying to force XMPP to act the way Wire expects it to. Otherwise why put all this work into Wire’s server and encryption in the first place?

So I think there are two big aspects of a potential Wire federatioin protocol to consider:

  1. How would the experience be for Wire-to-Wire users. I think with XMPP, this can be a 100% seamless experience. Wire can leverage as much of the existing XEPs as possible and potentially have a custom extension with some Wire-specific messages for extra features.

  2. What added benefit can be gained from using an existing federation protocol? This is where all your concerns seem to originate. Please note that with a Wire-specific protocol, of course you don't have these issues, because there is also no potential gain from federation outside the Wire server network. Since Wire's foremost priority is to serve Wire-to-Wire users, I think it can decide what stance it wants to take on chats with external users. Wire is free to require contacts to supports certain features. Encryption seems like the most contentious one. I'd be very glad if Wire would federate over XMPP even if they don't let users chat with non-OMEMO contacts.

I just want to also bring https://matrix.org/docs/spec/ to attention, it's in my opinion better than XMPP.

So when it comes to Matrix vs XMPP, I think there are several arguments to be made for XMPP. One is the extensibility built-in that allows an implementer of both server and client to extend the protocol with custom features as much as wanted without interfering with other clients/servers. But the biggest argument would be that it's 20 years old and still around and being used. That can't be said of any other instant messaging protocol. I agree that its flexibility and extensibility make it harder for users to know what to expect from a "modern" XMPP experience. As mentioned before, I think the lack of clients supporting these features is not due to the protocol, but due to the fact that all money invested in instant messaging goes to silo'd solutions.

So sure, it's probably easy to design a new chat protocol with exactly all the features that today's users expect. That's exactly what Matrix did IMO. But what if in 3 years the expectation is somewhere else. Are you going to re-invent another protocol called Tensor and start all over? Or what if the Matrix developers add some protocol features you don't want? What if you want an extra feature the Matrix developers don't want? XMPP's extensibility is proven. Leverage it.

stevenroose avatar Jun 16 '19 22:06 stevenroose

Sorry to intrude. There is a lot to read in this conversation, but I'm not clear on what the intended scope is of wire to XMPP integration.

Is this about XMPP as purely an inter-wire server federation mechanism?

Or is this about wire federating with XMPP in general?

haskellGardener avatar Jun 17 '19 15:06 haskellGardener

Is this about XMPP as purely an inter-wire server federation mechanism?

Or is this about wire federating with XMPP in general?

I think both. Inter-wire because the protocol is a good basis for any kind of federated messaging, and to other XMPP servers because more people being able to talk to each other across app borders is generally a great thing.

By the way, there are many examples of companies using XMPP for their own use case without federating with the larger XMPP network. Some of the largest user bases:

raucao avatar Jun 17 '19 16:06 raucao

Is this about XMPP as purely an inter-wire server federation mechanism?

Or is this about wire federating with XMPP in general?

As far as I'm aware, the discussion originated with Wire showing interest in implementing federation between Wire servers. So the primary goal is for different Wire servers to be able to federate with each other. The reason for this issue and other ones mentioning XMPP is just a suggestion to use XMPP for this federation work. While XMPP might be a good choice even if Wire is not interested in federating with non-Wire servers, my main interest is to see Wire servers being able to join the open XMPP network which would allow all Wire users of a federating server to chat with all existing XMPP users. (And more personally it would allow me to suggest Wire to my less technologically inclined contacts so I can chat with them over XMPP.)

I am not yet aware of any response from a Wire team member as to whether or not they are interested in federating in an open network beyond their own servers.

stevenroose avatar Jun 17 '19 16:06 stevenroose

So for inter-wire servers XMPP is an established technology that could be constrained to meet wire specific security requirements, etc.

For general XMPP federation is there a straightforward way to ensure that user messages have the same or similar security as asserted by the wire marketing materials, or is egress from the secure wire world a risk that users of wire may not realize until too late?

haskellGardener avatar Jun 17 '19 16:06 haskellGardener

For general XMPP federation is there a straightforward way to ensure that user messages have the same or similar security as asserted by the wire marketing materials, or is egress from the secure wire world a risk that users of wire may not realize until too late?

I don't know to the full extend what you mean with "security as asserted by the wire marketing materials". Basically when a Wire user adds an external user, it will provide a JID (Jabber ID) that is [email protected]. The XMPP protocol has messages to probe the server for the features it supports.

In the case of E2EE, the server just needs to support PEP storage which all modern servers do. Then, the Wire server can probe if the contact in question is using E2EE (A server supporting it doesn't mean that all users are using it.), this can also be done easily by querying the contact's server for the user's E2EE public key(s). When none are found the contact is not using E2EE and Wire could show a message "Your contact does not support encrypted messaging." or something similar.

Apart from E2EE, are there other features that you have doubts/questions about?

stevenroose avatar Jun 17 '19 22:06 stevenroose

My impression is that Wire has presented itself as being super secure. XMPP per se does not make guarantees like that. So general XMPP does not enforce much beyond perhaps encrypting the transport. So if users of Wire were to add XMPP users outside of the Wire system would there be any concerns that those users would be under the false impression that the XMPP non-wire users had a similar level of security?

haskellGardener avatar Jun 18 '19 14:06 haskellGardener

They do have a similar level of security, as long as they use OMEMO (or alternatively OTRv4 in the future). It's perfectly fine to not allow adding them as Wire contacts, if their account does not support a sufficient E2EE protocol.

raucao avatar Jun 18 '19 14:06 raucao

So there would be a means to recognize those clients that don't support adequate security and exclude communication with them if so desired.

Thanks.

haskellGardener avatar Jun 18 '19 14:06 haskellGardener

How is progress going on this? Or is it summer also in open-source land?

stevenroose avatar Aug 16 '19 11:08 stevenroose

How is progress going on this? Or is it summer also in open-source land?

We are not actively working on federation yet, as other topics currently have higher priority. Thanks for the discussion on this issue in any case, it will help us once we get to this topic.

jschaul avatar Aug 22 '19 11:08 jschaul

@L1cafe:

Or a web browser that cannot browse cleartext HTTP websites: It's not a web browser, it's an HTTPS browser, if at all.

FWIW all the major browsers have announced plans to enforce HTTPS, giving webmasters enough transition time to learn how to use Let's Encrypt (or another cert authority), and get their sites working properly with HTTPS. Some browser features already require HTTPS. Standard evolve, and extensible standard make that easier.

Matrix has already solved many issues of XMPP, like this precise E2E problem I am trying to highlight.

True, Matrix has built in E2EE but it's off by default on all the implementations I've used.

strypey avatar Oct 11 '19 10:10 strypey

@ jschaul:

We are not actively working on federation yet, as other topics currently have higher priority. Thanks for the discussion on this issue in any case, it will help us once we get to this topic.

Thanks for the update. I think the most useful way to focus discussion in this issue is for those who are familiar with implementing XMPP to have a skim of the existing Wire code and give a general overview of what would need to be done to implement:

  • a) XMPP federation between Wire servers
  • b) XMPP API between Wire servers and clients

Keeping in mind it's possible to do one without the other, they may have different sets of pros and cons, and it would help if developers experienced with XMPP could offer some dispassionate commentary, for both a) and b), on what would make XMPP a good fit for Wire and what its challenges or downsides might be.

EDIT: FWIW one advantage of implement the XMPP server>client API would be that existing XMPP clients could be used as a base for developing new Wire clients (branded or unofficial).

A separate "Implement Matrix federation" issue could be created to do a similar evaluation for that standard. If anyone has any other candidates - including create a custom Wire federation protocol - these could also have their own issues (or may already). Then a meta-issue like https://github.com/wireapp/wire/issues/266 could be used to compare and contrast them all, focusing as much as possible on technical pros and cons as they're exposed within each issue. This would provide the Wire team with a detailed technical resource they can use to guide their decision-making about how to implement federation in Wire.

strypey avatar Oct 11 '19 11:10 strypey

Update: internal discussions around federation (between Wire servers as a first step) are happening.

what would need to be done to implement:

a) XMPP federation between Wire servers b) XMPP API between Wire servers and clients

We have no plans to ever implement b) (to speak XMPP between Wire clients and Wire Servers.), or at least not in the next few years. So please don't focus your efforts on that part.

We plan to implement federation between Wire servers first. Whether that makes use of XMPP or not remains to be seen.

Your input regarding server-to-server communication (possible pitfalls, mistakes made in the past, etc) is appreciated. Please note that we plan to support MLS as the protocol matures, which has implications for group/conversation management.

jschaul avatar Oct 14 '19 14:10 jschaul

@jschaul thanks for the update!

If you have any questions about XMPP server-to-server connections, feel free to ask (I'm a server developer with >10 years experience in this).

Meanwhile a brain dump:

Protocol

XMPP server-to-server in a nutshell: do a DNS lookup, open a connection, authenticate using TLS, send/receive some stuff (you need to be able to accept incoming connections also, obviously).

Note: I strongly recommend ignoring the "dialback" protocol (XEP-0220) for any initial implementation. Although it used to be the primary means of server-to-server authentication 15 years ago, these days (almost) everyone has valid certificates thanks to Let's Encrypt, so the benefits of implementing dialback are low compared to the complexity it introduces.

So... if you go with certificate authentication only, the wire protocol is actually pretty simple.

You'll need access to an existing XMPP library with low-level APIs available. Most XMPP libraries are intended for implementing clients/bots, not servers, but often you'll find reusable components that deal with stream handling and JID normalization, that kind of thing.

I'm not a Haskell/Rust guy, but I know there is a small community of Rust folk in XMPP and they've been working on some stuff. I don't know how mature it is, but it's probably worth looking into if you're interested.

Developing your own XMPP stuff from scratch is not hard, but I'd recommend it only if there are no existing libraries available. You would need access to a namespace-aware streaming XML parser such as expat, a decent TLS library and a decent DNS library (capable of resolving SRV records).

Data model

That's the protocol layer that forms the core of XMPP - it's a essentially an authenticated pipe that you can send stanzas (XML packets) through. To interop with the rest of the XMPP ecosystem, you need to also consider what gets exchanged over the server-to-server connection, and in particular you need to figure out where the existing Wire data model maps to the XMPP data model, and where it does not.

For example, every user on a Wire server would need to be identifiable by an XMPP address (aka 'JID'), which is in the same format as an email address, i.e. [email protected]. Usernames cannot be longer than 1024 bytes in UTF-8 representation, with none of the forbidden characters listed in https://xmpp.org/rfcs/rfc6122.html#nodeprep-prohibited (again, existing XMPP libraries will generally have this stuff already). If your existing restrictions are tighter than these (e.g. ASCII only or whatever) then you're already :1st_place_medal:

If Wire supports multiple concurrent sessions from the same user, you will also need a unique identifier for each one (which you'd probably already have).

Presence

Presence, i.e. "status" is used to indicate to contacts when someone is online. If you don't have this concept, you can shortcut this by simply hard-coding everyone to online all the time. If you do have this concept, it's probably easy enough to map it to XMPP.

XMPP for IM leans heavily on the idea of a contact list (aka 'roster') for establishing trust relationships (at a basic level, who is allowed to see your status for example). If you have such a thing already it should be easy enough to map, if not then it's again pretty easy to fake it.

Messages

Messages are really simple. At their most basic they consist of a plaintext body, when not using end-to-end encryption. E2EE is a whole other topic which I won't drag in here (and you mentioned MLS, which the XMPP community is also tracking keenly).

<message to="[email protected]" from="[email protected]" type="chat">
    <body>Hello world!</body>
</message>

Now, here I think it's worth highlighting what appears to be a fundamental data model difference between XMPP and Wire. From what I can tell (please correct me if I'm wrong), all messages within Wire are within a 'conversation'. XMPP does not have any concept of conversations - it simply allows any entity to message any other entity. Group chats are an additional layer in XMPP.

Mapping this may be where most of the work lies. First of all, each conversation would need a unique address, something like [email protected]. That's the easy part, I see conversations already have unique ids.

Then you would need to make one of three choices:

  1. Implement MUC (XEP-0045) (the dominant group chat protocol in XMPP today)
  2. Implement MIX (XEP-0369) (a newer group chat protocol in XMPP)
  3. Implement something new using custom XMPP stanzas

MUC has the benefit of instant interoperability with a range of existing XMPP clients. The downside is that the protocol was developed back when IRC was a serious competitor, and it shows. MUC tends to assume that online users are in the chat, and offline users are not. There are effective solutions to make MUC more mobile-friendly than that, but it adds work.

MIX is a more modern alternative. It's simpler in principle, and is broken into a bunch of optional parts that you can choose to implement or not. It would probably map pretty nicely to Wire's current model. The downside is that today almost no XMPP clients support MIX (though this will likely change).

Finally, you could just implement something custom. This seems like the easiest approach while prioritizing Wire<->Wire federation. The downside is that you run the risk of essentially just developing something that looks like MIX but isn't quite.

Conclusion

There's a bunch of work here, but none of it terribly difficult (there would obviously also be a bunch of work involved in rolling your own protocol).

I applaud you for openly discussing things here, and not jumping blindly into rolling your own protocol as many do. I obviously recommend XMPP for its maturity and simplicity, but whatever you choose I'm personally going to be happy if it's open and documented. That makes bridges possible, rather than isolating Wire from other open federated networks and creating yet another closed system.

Again, happy to answer any questions you folk have.

mwild1 avatar Oct 17 '19 14:10 mwild1

@mwild1, awesome post! On MUC<>MIX, personally I think that, given that the MIX standard have been proposed quite some time ago and it's slowly being developed, new federators should leverage it if it helps them to integrate.

MIX was designed because MUC doesn't fit to the modern IM experience, so eventually we should all be using MIX in the end. So a player like Wire that would have to invest a ton of time into thinking about MUC compatibility should just implement MIX instead. It will be orders of magnitude easier and it will put some deserved pressure on the rest of the ecosystem to catch up on MIX implementation. It would be awesome to see a MIX-native client and server entering the network!

stevenroose avatar Nov 25 '19 16:11 stevenroose