[Proposal] Move out of Mediawiki

[aerostitch]

Hi,

The mediawiki website is quite old and automated crawlers think those are parked domains (example: http://duck.debian.net/static/sp/n/navit.html). Plus it might be a bit harsh for newcomers to find the informations right away. What I would propose is to migrate it to github pages in markdown, plug it to jekyll (the one that comes with github pages) and maintain all that in a repo like navit.github.org. I guess we could add sections to https://github.com/navit-gps/website I would be up to do the migration myself if you all agree on that.

Let me know what you think. Thanks for your feedback, Joseph

[sleske]

Could you explain why you feel the need to migrate? I like the fact that the Mediawiki site is easy to modify online, many people know the interface - what would migrating to Github pages buy us?

Your points about the wiki content being old and not always newbie-friendly is true, but that would not automatically get better by moving to GitHub.

[aerostitch]

Hi,

Could you explain why you feel the need to migrate? I like the fact that the Mediawiki site is easy to modify online, many people know the interface - what would migrating to Github pages buy us?

The main thing I could think of are:

• security:
  • Mediawiki, like any other CMS has several major security issues reported every months. If you want to have a look at the CVE related to mediawiki over time you can have a look at: https://www.cvedetails.com/product/4125/Mediawiki-Mediawiki.html?vendor_id=2360
  • The OS, the PHP engine, database engine and the webserver required to run this website also have regular CVE and need patching on regular basis.
  • using github pages means:
    • not having to maintain those and exposing less threats (you have people whose job is to maintain their infrastructure)
    • static-generated websites don't parse url arguments, don't accept POST, PUT, DELETE requests, don't have databases and run on read-only filesystems, regeneated with each commit you push to your repository, reducing the security risks by a huge deal.
    • the content is on a git repository meaning it can easily be reviewed using the usual PR system, just using a gh-pages branch
• ease to use:
  • I think about everybody knows how to write markdown and to push data to a git repo either by using the github web interface or using local tools and then push to the git repository
  • most people are used to the github wikis nowadays
  • As an infrequent contributor that sees a typo you don't have to have a login in a 3rd party to edit it, you just submit a PR
  • If you want to submit a change idea and get a review by several other people you can easily, again with a PR
• content-proximity:
  • In the wiki we have elements like coding style guides which generally have their place in a CONTRIBUTING.md file inside the repository it targets for easier use by contributors
  • the wiki pages on github are generally maintained by projects and can be accessed in a tab next to "Issues" or "Pull requests" when enabled, that's easier than to go on a totally separate website

Those are my main points. There are other, more subjectives ones, but that's probably be nitpicking. Joseph

[mvglasow]

My gut feeling tells me to stay with Mediawiki. (Same with the proposal to drop Trac in favor of Github tickets, which comes up from time to time).

First off, moving the content is a huge amount of work for which I would want clear benefit that is large enough to justify this. (I hope nobody is seriously proposing that we discard the last 14 years of work that went into the wiki and start from zero.) Currently, I am seeing the huge drawback that we would be moving from a very powerful system to a much more limited one. (There’s a reason why Wikipedia, Wikivoyage, OSM and others don’t host their content on Github wikis.) There are a couple of things Mediawiki can do which can’t be achieved with Markdown.

As for the Jekyll proposal—is the LineageOS wiki similar to what you have in mind? I find that one pretty confusing (and have not contributed since they moved over).

And finally, I am somewhat reluctant to move from FOSS to proprietary infrastructure (Mediawiki to Github) unless strictly necessary.

The points you mention are valid but I believe they can be addressed without discarding our present infrastructure:

• Security and patch management: This depends on the system infrastructure (mostly whether we have a dedicated server on which we can update packages as we like). A dedicated Linux server can be configured to install security updates automatically (optionally requiring manual confirmation for non-security upgrades), as long as packages are installed via package repositories rather than manually. For other setups we’d have to investigate if and how the same can be achieved. Maybe @pgrandin can shed some light on this; I believe he knows the server infrastructure quite well.
• Maintenance: I would expect the most time-consuming tasks being security updates and checking if everything is running smoothly. Both can be automated.
• Ease of use: I would expect the bulk of contributors to be equally familiar with Mediawiki and wikitext as they are with GH wikis and markdown. The visual editor now available in Mediawiki (usage being optional) is actually an argument for Mediawiki.
• Ease of use 2: Yes, currently we need a wiki account—presumably a legacy from the times when we were still on a Sourceforge SVN repo. I believe Mediawiki supports external authentication as well—we’d need to install it, but then people could log in e.g. with their Github, Google or OpenID accounts.
• Ease of use 3: Wiki edits are live immediately (unless there’s a review system in place, as is the case on some Wikipedia sites). With a PR, a casual contributor would need to wait for someone else to merge it.
• Content proximity: we currently do not have a CONTRIBUTING.md, but we do have a wiki page, so I don’t see a benefit here. As for navigation, I don’t see a separate domain as a problem. We could maintain a list of developer resources on our main website as well as on the README.md, which is how a lot of bigger projects (i.e. those who have outgrown GH’s wiki/ticket/pages infrastructure or were around already before these got introduced) do it. BTW, as for cross-referencing tickets or commits from the wiki, we have already solved that using Mediawiki templates.

[aerostitch]

My gut feeling tells me to stay with Mediawiki. (Same with the proposal to drop Trac in favor of Github tickets, which comes up from time to time).

Ok, I see that @sleske and you seem to feel strong about that and I've seen too many times personal tastes go in the way of contributing so I'll close those 2 proposals.

Just a few notes:

The reason I was proposing to move out of trac is (aside from the strong preferences I have) that every time I've seen people trying to maintain 2 bug trackers, it ended up causing more problems and discouraging people in general.

I hope nobody is seriously proposing that we discard the last 14 years of work that went into the wiki and start from zero

If you look at the work I've done this weekend you'll see that the migration can be done (and has been tested) without any loss. I can even import all the old tickets already closed from trac.

And if you look at the wiki migration proposal I'm not proposing to start from scratch but to migrate the content.

There’s a reason why Wikipedia, Wikivoyage, OSM and others don’t host their content on Github wikis.

Yes, generally because they have that since way before the static sites generator were created and (for the 2 1st ones) they are content management systems and unrelated to a project hosted on github.

As for the Jekyll proposal—is the LineageOS wiki similar to what you have in mind? I find that one pretty confusing (and have not contributed since they moved over).

No and the confusion is generally due to how people organize their content.

And finally, I am somewhat reluctant to move from FOSS to proprietary infrastructure (Mediawiki to Github) unless strictly necessary.

Not necessary, just proposing what I though would have been enhancements.

For your point on security and patch management I've never seen any serious sysadmin putting in place automatic updates. It's way too risky. But suit yourself if you want to go with it.

we currently do not have a CONTRIBUTING.md

Yes, part of the proposition was to add that as it's a common best practice on the opensource projects nowadays.

[pgrandin]

I would actually like to keep this discussion going for a bit. Here's my 2 cents from the point of view of someone who often has to deal with the wiki updates.

This is what a typical week of updates look like on our wiki:

A bunch of robot/spam users created, and some spam. Currently we get less than a handful of actual, useful edits per month. At its peak, we had over 400 spam edits in a week. Having to deal with this is a waste of time. We used to have manual account validation, which somewhat mitigated spam, but was taking even more time as many spam accounts were created manually by spammers and it's close to impossible to know beforehand if an account request is legit or not.

mediawiki doesn't provide a way to require validation from a trusted user before the content goes live, as the goal of mediawiki is to provide a platform where anybody can contribute. This works well for platforms that have a moderation team, but not for us.

Over the last 90 days, we had 57 accounts created, i'm sure that at least 90% of these are spammers. I was gone for a week and now we have 12 spam pages to clean up. Having reviews of the changes definitely makes sense to me.

We could use Github wikis, and we could also import the existing content as a readthedocs documentation. I actually toyed a bit with that idea some time ago: http://navit.readthedocs.io/en/latest/basic_configuration.html

The wiki format itself (a big collaborative content) is nice to allow people to share knowledge about Navit, but this freedom also makes it difficult to organize the documentation.

This was a quick reply to the thread, I'll think about it some more and post my thoughts.

[mvglasow]

@pgrandin Does the wiki currently use any of the extensions listed at https://www.mediawiki.org/wiki/Category:Spam_management_extensions?

Requiring NoCaptcha for user creation and adding external links looks like an obvious solution to make spam more expensive.

The StopForumSpam extension relies on an external blacklist to prevent spam edits.

The Nuke and SpamBlacklist/SpamDiffTool, while not preventing spam, allow for easier removal of spam (e.g. undoing all edits by known spammers, identifying all links to a particular domain).

The Moderation and Patroller extensions allow for review of edits, either after the fact or before they go live. Maybe we can get a few more people involved—I’d suggest giving that permission to everybody with write access to the repo.

The Social extension would be a way to rely on Github for authentication. I haven’t looked into ways to grant permission through Github (e.g. whoever has write access on Github also gets certain permissions on the wiki).

[jkoan]

Again we have to tons on Spam, exactly 126:

[zintor]

The spam is really annoying. Before moving out of mediawiki, I would like to read, if spam extensions are used or not. For now I am with mvglasow to stay on mediawiki, but spam extensions should be used.

[mvglasow]

Now that we have a mirror on Gitlab, we theoretically have the option of moving our wiki content there, which would change the “proprietary infrastructure” part. First question: would this work the same way on Gitlab?

As for usability, I might have been put off by the LineageOS wiki—judging by the instructions on their web site, this appears to be quite complicated. I don’t have any hands-on experience with @aerostitch ’s proposed setup, though. Is there any way we could test this (e.g. a POC or another project that uses a similar setup)?

[aerostitch]

Thanks @mvglasow I should be back home in a few days, I'll try to setup a POC on gitlab pages then. :) It seems to support more generators than Github and I'll have a look to see if there are direct export tools so that we'd have 1-to-1 pages in the 1st place. Current starting doc (just so that I don't loose the link): https://docs.gitlab.com/ee/user/project/pages/index.html

[mvglasow]

Any news on Gitlab Pages yet? I’d really love to see Navit moving to Gitlab; especially for tickets I feel it would combine the benefits of a free and open solution with a hosted one.

[aerostitch]

Hi, I unfortunately haven't had the time lately and will very unlikely be able to do that before March. Joseph

[mvglasow]

We’re in the first days of March… so just in case your schedule is getting more favorable, I have some input to contribute, as I transferred one of my own repositories to Gitlab today. Details here: https://stackoverflow.com/questions/54972004/move-a-repository-from-github-to-gitlab/54972151#54972151

In a nutshell:

• GitLab has an import feature for Github repos.
• Code and wiki are migrated 1:1, so are issues and PRs (even maintaining ticket numbers)
• Pages require some manual work; the SO answer details how to do that for pages that do not use a SSG

CI might require some work but in the end may be easier, as GitLab offers integrated CI pipelines. As a temporary workaround, we could still set up GitLab to push all new commits to Github, which would then trigger CircleCI.

[aerostitch]

Hey!

I thought I answered to that a long time ago, sorry. @pgrandin has started to move the wiki pages to readthedocs and I pushed several ones too. I takes a lot of time as it's 1 at a time and I update some of them and reorganize a little but you can see the beginning of the result here: https://navit.readthedocs.io/en/trunk/index.html

Would that format fit your needs for now @mvglasow ?

Thanks Joseph