Start deprecating old VCS support

[humitos]

Currently, we support the following VCS in our platform:

• git (git)
• bazaar (bzr)
• mercurial (hg)
• subversion (svn)

I'd like to start a conversation about the possibility to deprecate the ones that are mainly not used. In particular, subversion. I went ahead and took some numbers from our platform to find out how much each of them is used:

Number of projects using VCS

• git: 247,599
• hg: 1,773
• bzr: 1,765
• svn: 1,673

Number of projects using VCS with a build in the last 30 days

• git: 14,141
• hg: 46
• svn: 31
• bzr: 12

Number of projects using VCS with a successful build in the last 30 days

• git: 11,532
• hg: 37
• svn: 2
• bzr: 0

Also note that I'm not discarding spam projects. So, these numbers are lower.

---

I'd propose to start removing svn (and probably bzr as well) from the UI to avoid new projects being imported using these VCS. Then, the next step would be to communicate projects that are not spam that we are deprecating support for the VCS they are using.

Also note that to select svn and/or bzr users have to import the project manually, which gives them a worse UX of the whole platform: manual import, manual webhook setup, etc.

This will allow us to not maintain features that almost nobody uses but consume our time in different ways.

Edit: fix numbers that were wrong (missed a DISTINCT in the query)

[agjohnson]

It would be helpful to know when these projects were created as well. That is, were bzr and svn projects mostly created 5 years ago? Git is obviously the primary vcs, but I would guess that new projects are even more leaning towards git. Also might be worth getting this information on hg as well.

I'm :+1: on deprecating support on svn and bzr. Slowly helps here: removal from the UI, then removing actual support in the backends. Though, given above, I doubt we get a lot of new projects coming in with either.

[humitos]

Notes from our meeting in January 😄 :

• Make a deprecation timeline (Remove from UI at this time?)
• Find projects to email (Valid build in last year) & email them
• Finally, deprecate/remove code

[agjohnson]

More notes: maybe it makes sense to scope this down to removing suggestions for bzr/hg in our docs, and point users to always use git. Code removal will be a ways out

[humitos]

I'm fine with removing this from the documentation as the first step. However, I think we are always pushing back on deprecating things properly with a hard final date. Finally, we end up with inconsistencies between our docs, our code, and what we want. I think we should start becoming more strict with deprecations, in particular with those that are not heavily used.

[agjohnson]

We could definitely add a bit more to the scope here. We get the most value here out of consistently pointing new users/projects to Git. The main point is that full removal and deprecation is going to be a fair amount of work and we won't get a whole lot of value out of this work.

As a big chunk of work, this doesn't yet have priority on our roadmap. But we can prioritize small steps in this direction more immediately.

Full deprecation and removal of this code will require UX/UI around unavailable VCS, contacting users, monitoring deprecation before code removal, etc. So, quite a lot considering we don't spend a lot of time with the code now anyways.

[humitos]

@agjohnson

But we can prioritize small steps in this direction more immediately.

We already have done this in our meeting from January https://github.com/readthedocs/readthedocs.org/issues/8840#issuecomment-1144879129

Full deprecation and removal of this code will require UX/UI around unavailable VCS, contacting users, monitoring deprecation before code removal, etc

I'd expect that the new UX/UI does not support these old VCS at all.

On the other hand, having a hard date of removal/end support communicated to users removes the requirement of a "monitoring" step.

So, quite a lot considering we don't spend a lot of time with the code now anyways.

I don't fully agree with this sentence. It's true that we don't touch this code too much, but simple things like #9424 require a lot more thinking, testing and QAing.

[agjohnson]

We already have done this in our meeting from January

Yeah we had some direction on this from that meeting, but in our roadmap planning call, this felt like an issue that we should bump out a quarter as it didn't really align. But we did agree we should have something from this on our radar.

That said, I think we're just describing making "Make a deprecation timeline (Remove from UI at this time?)" a few discrete steps. We can prioritize some smaller actions first

I'd expect that the new UX/UI does not support these old VCS at all.

Disallowing new project VCS choices could be another more immediate step.

On the other hand, having a hard date of removal/end support communicated to users removes the requirement of a "monitoring" step.

Though we will want to watch things periodically either way, we can expect new inbound support as we start the deprecation. We probably won't be lucky enough to get away with just emailing users and then returning to delete the code, there's some lump of work in the middle.

It's true that we don't touch this code too much, but simple things like https://github.com/readthedocs/readthedocs.org/pull/9424 require a lot more thinking, testing and QAing.

That's fair, there is work hiding here. At least for this next quarter, we hope to be touching the backend code less than we did in Q1/2, so this might not be a driving function of this change.

[humitos]

It would be helpful to know when these projects were created as well. That is, were bzr and svn projects mostly created 5 years ago?

This table shows the trending pretty well:

Screenshot from https://ethicalads.metabaseapp.com/question/238-projects-grouped-by-pub-date-month-and-vcs-type-success-build-in-last-year

This plot shows "projects with at least 1 success build in the last 12 months grouped by VCS type"

Screenshot from https://ethicalads.metabaseapp.com/question/239-projects-vcs-types-with-a-success-build-in-last-year

[humitos]

Some notes from the meeting that I was able to write down. Feel free to edit them or add context, comments.

• soft deprecation
• remove mentions from bzr and svn from all the documentation
• keep the code around
• avoid projects to select bzr and svn in the UI or similar places
• define a deprecation process to finally remove the code and break compatibility (see https://github.com/readthedocs/meta/discussions/46#discussioncomment-3587302)

[ericholscher]

I think we could even get rid of hg in the docs and drop downs. I like simplifying to be able to just support git would dramatically improve a lot of our messaging.

[benjaoming]

remove mentions from bzr and svn from all the documentation

Also "VCS"?

Normal git users don't go around saying "VCS" no more.. it's like VHS :)

We put "VCS" in a bunch of navigation labels, links and titles.. it's quite bad, I think people won't get what an entire core section of our feature list is about.

[ericholscher]

@benjaoming What would replace it? Just saying git?

[benjaoming]

In some cases "git" is enough, some cases it can work with "git platform". I'm not convinced that "git host", "git provider", "git system" or "git service" sound good.

[agjohnson]

I'll bump this to Q1 assuming we're doing a soft deprecation on providers to start. That seems achievable

[benjaoming]

@agjohnson

Could a deprecation look like this?

1. Remove mentioning of bzr, hg and svn from documentation (or archiving them in a legacy section)
2. Once our new Policy for Supported Tools is added, then update it: https://github.com/readthedocs/readthedocs.org/pull/7859
3. Remove support of bzr, hg and svn from import wizard
4. Notify remaining projects
5. Finally remove build support, API support and clean up code + API docs

I am asking because I'm interested in how soon we can start to rename and simplify some of the current VCS articles, since they are going through some processing over the next weeks.

[agjohnson]

I think what we're talking about above is not doing 4 and 5. Doing all of the work to notify projects and break their usage will be a fair bit of work, and might be overall negative value too.

I know that having the VCS code around is still a bit of a liability. It would be great if we could version our build backend API in some fashion and disregard this code going forward. Perhaps if that is a possibility that would be the point in time to remove the code.

[benjaoming]

@agjohnson

Sounds good - do you have any objections about moving forwards with the first item during the coming weeks? It's mostly about a move towards saying "git" rather than "VCS". Regarding actual contents, we don't have much more than this VCS Support Matrix to remove: https://docs.readthedocs.io/en/stable/versions.html#version-control-support-matrix

[agjohnson]

do you have any objections about moving forwards with the first item during the coming weeks?

Nope! It's something we have to do at some point. I would say you should dictate when it makes the most sense to change this, as you'll be changing the most around documentation and we should avoid derailing your refactor work with content changes as much as possible.

[benjaoming]

Great! The work has started here: https://github.com/readthedocs/readthedocs.org/pull/9675

[humitos]

We merged the PR that disallow other VCS that are not Git for new projects. Another step forward to accomplish with this issue 💪🏼

[benjaoming]

We're removing this one in https://github.com/readthedocs/readthedocs.org/pull/10186 - I think we are on a good track :)

[benjaoming]

Two things that I noticed while working with the Git backend:

1. It would be great to get rid of the other backends (hg/bzr/svn) in order to start working on simplifications after #10430

2. For the deprecation announcement, we can tell projects that export to Git, that they can edit their repo URL instead of creating a new project :bulb: I'm not sure that we can fully support all aspects of editing a repo URL from for instance hg to Git. However, I think we can encourage people to do this in order to keep all their old builds (and not lose their project slug to an automatic deletion). We should be able to pull some tricks such that versions built and published with the previous VCS are kept. This may also be an option for a 5 year old project that doesn't intend to do new builds, needs to keep the old builds and wants to avoid getting deleted.

[ericholscher]

@benjaoming I'm not sure why we need to do anything specific when projects change their repo url to keep old versions? That should be the default behavior... Is there a benefit to deleting and re-creating the project vs. just changing the repo url that makes you think we'd suggest deletion?

[benjaoming]

@ericholscher if they export their project from X to Git, then they might:

a) be correctly exporting tags and branches :heavy_check_mark: b) or they might not export tags and branches :warning:

If a), then I think we are safe, but it's only by skimming the code: If the export is fine and all tag and branch data has the same names as before, then it shouldn't matter that the commit hashes change.

If b), then I think our behavior is to auto-remove Versions that are no longer matched by any branch or tag? I think we have the Feature flag SKIP_SYNC_VERSIONS because of such issues?

One question that I still have for a) is if we risk running builds again when we see a tag/branch again in sync_versions_task, but I think it's only when the stable version is bumped that we actually trigger builds from here?

---

Edit: It's possible to argue to rebuild old branches/tags in order to get things like "Edit on GitHub" to link correctly. I'm just pretty skeptic that rebuilding a bunch of old branches and tags will work in practice for most projects... could say that projects using svn/hg/bzr probably already suffer from legacy issues :)

[ericholscher]

@benjaoming This seems like an edge case and I'm not super worried about it, since I don't think many projects will go down this path. I think users will just rebuild versions if they need them, otherwise it will just work? 🤷

[benjaoming]

@ericholscher I'm trying to summarize my understanding of how this looks from user POV. Does this draft make sense?

We are deprecating support for SVN/Mercurial/Bazaar and focusing our efforts on Git.

To continue building your project on Read the Docs, you will need to provide a Git repository.

1. Export your SVN/Mercurial/Bazaar project to a Git repository.
2. If your documentation has multiple versions, make sure that your tags and branches are re-created in Git, as they will be matched with existing versions on Read the Docs.
3. Edit your Repository type and URL in your Project Admin
4. Save :heavy_check_mark:

After saving these changes, you can continue setting up automated builds with your Git provider by following these steps: https://docs.readthedocs.io/en/stable/guides/setup/git-repo-manual.html

In case your project has multiple versions that you need to either update or recreate, then you can re-build a specific version in your Project Admin's "Builds" section.

[humitos]

I don't think that people using SVN/Mercurial/Bazaar will do that. If they are using those VCS it's for a (good/bad) reason and they just don't want/can to use Git. So, telling them to export/convert their repository into Git just to use Read the Docs is a no-go, to me.

I think we just want to communicate these users that these VCS won't be supported in the future and if they want to use Read the Docs, they have to provide a Git repository with the documentation. Will they do that? How they do that? I think it's not our business 🤷🏼

Pointing back to this comment again: https://github.com/readthedocs/readthedocs.org/issues/8840#issuecomment-1222558652. We have just a few projects (238, in total --some of them may be SPAM, testing/demo projects, or just invalid) using deprecated/unsupported VCS.

[ericholscher]

Agreed. I think we just alert these people, but we don't have to do any work or thinking about how they migrate. It's up to them to decide what is best for their project.

[humitos]

We have 6k total affected projects (including spam and everything there). I'm targeting the fully removal of the code for November 7th. I've already started writing the email we will be sending to communicate this following the standard process we've been using for a deprecation like this one.

[ericholscher]

@humitos I know we discussed starting with bzr & svn, since hg is still somewhat used. Are you wanting to remove everything?