operations
operations copied to clipboard
openstreetmap
Federating OSM communities' rooms through OSMF-hosted Matrix servers
Suggestion: Federating OSM communities' rooms through OSMF-hosted Matrix servers
Volunteering to help: Atrate and Natrius
Background:
One of the principles of OpenStreetMap is decentralization, which can be noticed both on the mapping level as well as on the communication level. Many countries (local groups and local chapters) have their own communication methods, be it IRC, Discord, Telegram, Slack, Matrix, Mail or anything else. This plays into the principle of decentralization very well and provides many benefits, but it can also have some drawbacks, like a lack of participation in communities due to fragmentation and a lack of a "common ground" for OpenStreetMap's communications worldwide (though the OSM World Discord can serve as such).
It's worth noting that France is using Matrix for internal communications. Mozilla has also recently switched to using it and Germany's Bundeswehr is thinking about using it as well.
To serve this problem of fragmentation is a tedious task, especially while trying to preserve the decentralized nature of OSM communications. Keeping that in mind, there is a way to make everyone happy – reduce fragmentation and increase global participation while keeping decentralization. This solution is the Matrix protocol – a decentralized messaging protocol that has immense potential for federation, with well functioning bridges for popular services like Discord, Telegram, Slack,IRC and even e-mail, which could be connected to the mailing lists, too. A sizeable amount of official bridges can be found here: https://matrix.org/bridges/
Proposal:
An official OpenStreetMap Matrix server (Synapse) could be set up to host the Matrix rooms and community. This should not require too much on the resource side of things, due to the way the Matrix protocol works – users will be able to join rooms hosted by OSM through other homeservers, such as Matrix.org and many others. This would also allow us to host bridging services (bots) on the OSM servers instead of using public bridges, reducing latency significantly.
Through such bridges, the plethora of OSM rooms across different services could be connected seamlessly on a single platform while keeping their independence and not centralizing power over the rooms themselves.
Volunteers
Atrate - I have quite an amount of practical knowledge maintaining and bridging Matrix rooms and communities (both unencrypted and encrypted) :) I also have some amount of experience managing Linux servers (but haven't deployed a Synapse server before).
Natrius (Negreheb) - I wrote a installation-guide that was once the official guide for installing the synapse server for quite some time until they reworked their own on github, created my own instance and have some experience managing my own Linux server.
This proposal has been written after a discussion with the CWG and the LWG.
I'm part of the OWG. I am fully in support of us having an OSM instance of something like Matrix.
@Atrate @natrius Could you detail what sort of hardware we'd need to run Matrix properly? And for High Availability? Would it be better if we run it on Cloud (eg: AWS) so that we could use services like AWS RDS and AWS EFS?
Storage (and, roughly, bandwidth requirements) can be calculated using this spreadsheet:
https://docs.google.com/spreadsheets/d/1clrE4WFT7A5NS5AJUdU-mbDZ9rS9fzl6QEbUFJezo7U/edit#gid=0
Some useful info about Synapse can be found in its GitHub repo: https://github.com/matrix-org/synapse
I've asked on #synapse:matrix.org about the other HW reqs, will update this comment if I get them.
I have setup synapse before, though on Fedora and I don't know offhand what the situation is for Ubuntu.
offhand what the situation is for Ubuntu
The package is matrix-synapse. 20.04 has 1.11.0-1. 18.04 has 0.24.0. There are no backports yet. Looking at Debian there is a backport, so it makes me think we could get a Ubuntu backport, but it's probably easier to wait until 20.04.
This proposal has been written after a discussion with the CWG and the LWG.
As per previous discussion it would also meet the needs of the SOTM WG.
@Atrate What do you recommend? The docker images used by https://github.com/spantaleev/matrix-docker-ansible-deploy seem to me the best option?
I never used docker and i won't. I wrote my initial synapse-install-guide for ubuntu server 18.04 with postgresql.
Using docker would be quick and simple, but if we have a couple of experienced people, @tomhughes and @natrius we can set it up from scratch with Postgresql, will be cleaner probably.
EDIT: I've taken a look at these docker images, though and they seem pretty damn good, though. Pgsql by default and easy TURN set up seem like a great convenience factor.
It also seems to include quite an amount of optional bridges so while some people might not like docker, it would lessen the OWG's workload significantly.
I operate a small matrix homeserver for a few people, no docker involved (I got some ansible playbooks for that). Even with larger rooms, the DB and the media directories don't grow that much. I run the PostgreSQL on ZFS, which even gives ~2.9x compression on the DB.
The bridges, well, they are mostly not that great. The IRC bridge works reasonably, but we could rely on the matrix.org bridge for the beginning, as rooms are replicated onto all involved servers anyway.
My installation uses python packages, straight from pypi, so updates can be rolled out quickly.
Aside from the technical discussion, we'd need some policy decisions as well: Which domain would be used? Who should be allowed to register an account?
Aside from the technical discussion, we'd need some policy decisions as well: Which domain would be used? Who should be allowed to register an account?
I'm not sure if I'm one to suggest policies, but I guess that a good, simple domain could be chat.openstreetmap.org for the homeserver registration URL and just openstreetmap.org for the displayed user account URL.
To keep it cleaner and easier to manage, maybe only people involved with the WGs or with managing the rooms themselves could be allowed to register an account, this way anyone with an xyz:openstreetmap.org account would be easily verifiable as someone trusted enoguh.
To keep it cleaner and easier to manage, maybe only people involved with the WGs or with managing the rooms themselves could be allowed to register an account
I think many people wouldn't use the OSMF matrix server for their account, because they already have an account somewhere else and having multiple accounts is kind of weird in Matrix (maybe that'll change). Nevertheless we should be able to point people at our server, with our terms & conditions, so they can sign-up easily. Having to tell them "go to a trusted WG member to get an account" would be a dealbreaker since the vast majority of active OSM contributors is not part of any WG.
We also have to think about whether we want to host Riot as well.
I guess it would be possible to talk with the guys in the synapse channles or some guys on their github to get some support for a kind of Single-Sign-On, so everybody who wants to use the osm-matrix-instance is able to use without extra logging in, for example. If this is happening (or is possible), i would not allow extra registration, to be honest.
and i would go for a @username:openstreetmap.org approach as well, without any subdomain. Better to tell someone, better to find someone and clear.
and i would go for a @username:openstreetmap.org approach as well, without any subdomain.
Fully agree.
support for a kind of Single-Sign-On
There is no OAuth support in Synapse at the moment. There is only an unmaintained (as so many 3rd party projects in Matrix) REST auth module, which we would need to customize to run against osm.org.
Also, I believe there is the issue that not all OSM usernames are valid matrix user IDs.
Could you detail what sort of hardware we'd need to run Matrix properly? And for High Availability? Would it be better if we run it on Cloud (eg: AWS) so that we could use services like AWS RDS and AWS EFS?
The hardware requirements are modest and only really get bigger if a lot of users are using it. Just to tell some numbers, for the start 4 cores and 8 G RAM per instance would suffice (though, synapse will eat all available memory). Synapse can be run easily behind a reverse proxy (I use nginx), it needs PostgreSQL, but someone in OWG should have some experience with that ;). It can use either the filesystem or S3 for media storage.
If we don't want to put hardware into racks, we could start off with e.g. Digital Ocean:
- Synapse: 2x 4 vCPU, 8 GB RAM: $40/mo each
- Postgres: Hosted 2 vCPU, 4 GB RAM, 38 GB Storage with a standby node: $100/mo
- S3: $5/250 GB storage
Having to tell them "go to a trusted WG member to get an account" would be a dealbreaker
Oh, I think I worded my previous comment ambiguously.
What I meant was that the xyz:openstreetmap.org addresses would only be available to WG/OSMF members, so thay they could be easily recognizable and trustable.
hi - in terms of SSO, Matrix can do SAML or CAS out of the box; we can probably help with OAuth if that’s you’re only option. In terms of ubuntu packages, you may want to check out https://matrix.org/blog/2020/04/06/running-your-own-secure-communication-service-with-matrix-and-jitsi
It might be beneficial to consider using Dendrite instead of Synapse, when it becomes stable, as it's a second gen Matrix homeserver implementation.
https://github.com/matrix-org/dendrite
It might be beneficial to consider using Dendrite instead of Synapse, when it becomes stable, as it's a second gen Matrix homeserver implementation.
https://github.com/matrix-org/dendrite
Or Construct for that matter, which is an already working, currently trusted users only implementation, using fraction of memory and disk compared to Synapse. But it's not yet a viable option (but it works).
Also there is a Rust based server in development (Conduit) which is promising but not yet ready for the show.
Operations-wise running a matrix server (Synapse or Construct) is pretty hassle free (my sever has about 100 users), and hosting a RiotWeb is basically serving static files. I would be pretty happy to see OAuth2 work, I guess it's not impossible, especially if NV throws some human resource on it.
As a sidenote I am using the "real" Debian packages for runnin Synapse (not the NV provided ones since they're using non-standard paths and duplicated libraries), and had no real problem with them. Converting it into the workers (multiprocess) running is pretty straightforward too.
Domain-wise I do not see any hard problems since it would run on "openstreetmap.org" I guess, and what the specific server is called is up to taste and the well-known/DNS SRV redirection.
Also I second the opinion that gateways are not great to have in an otherwise Matrix using community as the supported features of the different networks make conversations hard, or enforces problems of other networks onto matrix [like joining irc through matrix.org supported gateways results people being kicked out periodically due to ircops wishes]. Preferably people should really use matrix and not scattered in various services.
You can ping me at @grin:grin.hu over there. Also there are live communities already, which can be observed on +openstreetmap:matrix.org community or by searching.
Quick heads-up: Riot (the most used Matrix client) has changed its name to Element: https://element.io/blog/the-world-is-changing/
I'd like to throw in my hat to any help pertaining to moderation, etc. I'm a moderator for a few Monero (another FOSS project) related Matrix rooms that are bridged to IRC, but I've also contributed a good amount to OSM via membership donation, organizing mapathons, things of that sort.
The membership working group also discussed using a hosted matrix instance, to untangle some privacy issues for us.
I would have a look at a hosted instance. https://matrix.org/hosting/ It does not look that expensive, and would let our volunteers focus on more mapping related tasks. However I am not in a position to push that too much. By chance one of the providers is ungleich, which I have supported financially and gotten some shares in return. A conflict of interest.
Just to clarify - what exactly is being proposed here? Merging Discord, Telegram, Slack, IRC, Matrix, mailing lists etc into one megachannel so that what posted in say IRC is replicated to Discord and mailing lists?
Or backbone for read-only replication (similar to existing read-only channel replicating for example mailing lists)?
A merging of multiple communication channels into such a "megachannel" would obviously be an impossible to moderate mess. The most important part of this proposal is having OSMF-backed Matrix channels on an openstreetmap.org domain.
Then, select communities would be able to voluntarily bridge one or a couple of their channels with common channels on Matrix and unbridge them on demand.
Obviously, bridging mailing lists in a read-write way would be a bad idea. Certain channels for e.g. OSM news feeds and the mailing lists could be created in a read-only manner, so that they can be bridged to other communities as a sort of "news feed".
A little bit like that. Currently there is IRC germany and Matrix germany combined, same for austria. Have not checked for others. OSM Community and Local Chapter group has an bridge with telegram and IRC.
Create the OSM Server where we can integrate bridges that are needed. This way there could be a room with bridges to Slack, IRC, Telegram and Discord. So far i would not integrate other channels and for the beginning i would start with creating an official server, combinind there all already existing matrix-channels. Some of them even have bridges to some of the mentioned services already.
I would not(!) include Mailing Lists there. A Mailing List is not a chat and i guess a lot of mailing list users would rightfully speak out against this idea. A special channel solely for that purpose 'could' be possible.
I think a mega channel is possible. The PINE64 community has a very active mega bridge: https://wiki.pine64.org/wiki/Main_Page#Chat_platforms
