Improve redirects feature

[agjohnson]

trafficstars

We brought up the idea of using our YAML file for redirects. There are a number of problems with this though, as well as many existing problems.

Current problems:

• ~~If the application is down, redirects don't work~~ Update: still technically true, but the application has been super solid
• ~~If we wanted to make a Azure storage hosted version, we don't have an application or ngnix to fall back on for redirects~~ Update: Proxito is the layer in front of S3 storage now, and applies hosting :magic_wand: up front
• Redirects are project level currently -- this is still a large issue

New problems:

• Keeping per-version redirects in YAML makes a lot of sense, but per-project redirects are still valid and do not make sense in YAML (We discussed this in depth at https://github.com/readthedocs/readthedocs.org/issues/8811)
• Because of ^ we need 2 sets of redirects -- the UX would probably be to make current redirects "Project Redirects"
• ~~Making per-version redirect model is okay, as we can update the database per-version on build, but the point above about azure redirects is valid still.~~ No it's not!

A more current description of where this work is:

• We need to finalize a design decision on how we're technically exposing project level redirects in a configuration file
• There are a few extensions folks have authored to integrate with our API, so this could also be a path forward, though it does feel like a native feature is long overdue here

[humitos]

Some things have changed since this issue was created. I think it's a good moment to revive this conversation. The most important decision to make, IMHO, is the scope of the redirects: project vs. version.

I'd vote to have everything in the config file and de-prioritize any web UI for this work. I do see these options we could implement redirects on YAML file:

• only respects redirects from the default version YAML file and ignore any new redirect created in other versions (including PR) --similar to what we have now, but changing the web UI by the YAML in the default version
• give priority to redirects from the current version requested, and fallback to project-level redirects

[ericholscher]

Yea, I think we can follow a similar approach here, where we can have project-level redirects defined, and we use the default_version's project-level redirects. That at least gives us a specific way to make it work, and lets us keep it in YAML at the version-level which would be nice.

[pradyunsg]

Ahoy! This would be great for pip, and anything to have these redirects allowed at the "root" level would be ideal.

We'd like to start using this for things like error messages and survey links, to make them easier to reference and to have them look more official. This is especially useful for things like https://pip.pypa.io/surveys/backtracking, which are really nice. :)

[ericholscher]

@pradyunsg I don't think you need it in the config file for this? You could set it up on the RTD admin. I'm 👍 on doing more around redirects tho.

[pradyunsg]

We've got those configured in the admin pages (that's why that link works 😉).

It'd be nice to have this be something that we can have under version control and accept PRs for (like a config file)! That'll help make it easier for contributors to help add to this, and all the advantages that come with that. :)

[pradyunsg]

I just noticed that I didn't make a note of why I brought this up again: I'm doing a rewrite of pip's documentation basically, and this would help make that easier to review for folks + to roll out those changes.

Additionally (as I mentioned) we're starting to use this for "official links" in tweets / error messages etc. That means that the project-owners become the blockers for contributors' PRs that need things like this which... isn't ideal? (One of our contributors even filed a PR for trying to add a page with a client-side redirect, because none of us had come around to doing this on the server side: https://github.com/pypa/pip/pull/9680).

[pradyunsg]

Okay, so... as it turns out, I'm very interested in this. What can I do to help move this forward?

[ericholscher]

@pradyunsg The big questions are around how to make the UX make sense. What are project-level vs. version-level redirects, how do we handle them both, etc. I think in the past we've had project-level stuff uses the default version, but I don't love that since it's pretty confusing and non-obvious to users. If we end up with 2 different Redirect UI's, it gets really wonky as well.

[agjohnson]

Just a note: while we're figuring out how to best make this a native feature, there are a few options for authors here:

• https://pypi.org/project/readthedocs-cli/ -- this might be what we effectively end up building into RTD, a post build step that updates redirects from a configuration file.
• https://documatt.gitlab.io/sphinx-reredirects/ -- meta redirects
• https://github.com/wpilibsuite/sphinxext-rediraffe -- meta redirects

[ericholscher]

We had a lot more discussion about this here: https://github.com/readthedocs/readthedocs.org/issues/9188 -- I think we can probably close this as a dupe, but not 100% sure they are solving the same things.

[agjohnson]

Yeah one of these should be closed probably. I believe we landed on not implementing this in the config file, at least to start.

[pradyunsg]

Thanks for the confirmation on that @agjohnson! I've ended up implementing this via a script for pip in that case; that uses the RTD API. :)

https://github.com/pypa/pip/blob/main/.readthedocs-custom-redirects.yml https://github.com/pypa/pip/blob/main/tools/update-rtd-redirects.py

[stsewd]

Some related issues:

• Faster redirects: https://github.com/readthedocs/readthedocs.org/issues/10112
• When do redirects run?
  • https://github.com/readthedocs/readthedocs.org/issues/10314
  • https://github.com/readthedocs/readthedocs.org/issues/10049
  • https://github.com/readthedocs/readthedocs.org/issues/8309
• Import/export https://github.com/readthedocs/readthedocs.org/issues/7113
• Redirects in other domains (external/canonical/etc):
  • https://github.com/readthedocs/readthedocs.org/issues/8250
  • https://github.com/readthedocs/readthedocs.org/issues/9614
• Others: https://github.com/readthedocs/readthedocs.org/issues/9240

Eric also mentioned having a way to have more specific redirects per domain, I like how CF does this (*.rtd.com/, */foo). Haven't thought much about how we will be able to support these, but at least matching things per-domain could work.

[ericholscher]

Yea, I think https://github.com/readthedocs/readthedocs.org/issues/9614 is part of what we hit when migrating EA domains. It would be nice to be able to apply a redirect to a domain without it effecting all domains (like PR builds), so that you can redirect an old domain cleanly. For EA, we wanted a ethicalads.io/* to www.ethicalads.io/$1 redirect, but couldn't get it working.