[NEW] Rolling downgrade, forward compatibility

[zuiderkwast]

The problem/use-case that the feature addresses

After a rolling upgrade of a cluster, if some problem is found, such as CPU or memory usage or some bug, the administrator wants to be able to a rolling downgrade again and take some time to investigate the problem.

When adding new nodes to a cluster, the standard procedure is to add new nodes as replicas to existing nodes. For replication to work, we require that a replica's RDB version >= primary's RDB version. This works for rolling upgrade, but it doesn't work for downgrading if the RDB version has changed.

Some system that I've been made aware of is stuck on Redis 6 (RDB 9) because of the requirement to be able to do rolling downgrades.

The cluster bus is already backward and forward compatible AFAIC. It would be good to avoid breaking that. :)

Note that we're only talking about adding new nodes of an old version to a cluster. We don't need to start an old node with a config file from a new version such as nodes.conf, nor do we want to reuse a node-id when replacing nodes in the cluster.

Description of the feature

Possibility to generate RDB in an older version than the latest, if no keys require the newer RDB version.

An example is the bump from RDB 10 to 11 (that I'm guilty of by introducing listpack encoding for sets). If we configure the node to generate RDB 10, the node can easily be made to store this listpack as a RDB_TYPE_SET (which is just each element as a string) instead of RDB_TYPE_SET_LISTPACK (a listpack dump). The bump from 9 to 10 is similar.

Alternatives you've considered

#59 = Full sync without RDB preamble, although it could be combined with a REPLCONF to check if the replica supports the current RDB version and fallback to AOF without preamble. (RDB is a bit faster and smaller in size.)

Additional information

Redis never supported downgrades. This would be a Valkey feature that would make Valkey superior in terms of compatibility.

Valkey/Redis	RDB	Why bump
Redis 5.0	9
Redis 6.0	9
Redis 6.2	9
Redis 7.0	10	Change ziplist to listpack for hashes, sets, etc.
Valkey 7.2	11	Add listpack encoding to sets
Valkey 8.0	11

[madolson]

The major decisions:

1. Do we want to be able to support this downgrade.
2. How far back to we want to support restoring into older versions? (At least one version, maybe much further)
3. How will we document this.
4. What is the format we will use to support forward compatibility. One option is to have the primary send an older RDB compatible object. Another option is to send it in AOF format.

Consensus from core meeting seems to say yes for 1.

[madolson]

We also want to create a separate issue outlining how we can add guardrails to prevent usage of new features which would break forward compatibility. The discussed example was for hash field expiration, which must necessarily change the RDB format.

[zuiderkwast]

A pre-condition we seemed to agree on is that a downgrade is only possible if none of the new features in the new version are used, for example a new key type that doesn't exist in the old version.

It seems pretty strait-forward to generate RDB of an older version. How to select the RDB version to output? There are some possibilities:

1. A new config rdb-version, default to "latest". The admin needs to set it to the required version before connecting the downgraded nodes.
2. Autodetect which RDB version to use. Since Valkey 8.0, replicas send REPLCONF version. If they don't, we can assume they're using a very old version, so we can pick let's say RDB 9, and we can use this for the full sync. There are some corner cases here though:
   • What if a new replica is already performing a full sync using the latest RDB and a downgraded replica connects? It can't share the already created RDB dump. Shall we create a new fork to do a new RDB dump with RDB 9, or shall we return some "Try again later" error?
   • How to handle RDB dump on disk? Which version should they use?

Option 1 (using a config) seems like the easiest and most explicit way.

[soloestoy]

Some of my concerns:

1. From the perspective of "official support," honestly, I don't really recommend supporting this feature. The official stance represents authority and standards, and I'm worried that once the downgrade feature is incorporated into the standard, it might be misused. Although our original intention is to allow rollbacks during upgrade issues, users could potentially use it in other scenarios, which is a common occurrence.

2. The development and maintenance cost of supporting downgrades is considerable. Based on the discussion above, if we are to be backward compatible with old RDB formats, it means retaining the code for generating old formats. A few versions might be manageable, but in the long run, the burden could be very heavy.

3. We haven't considered the aspect of incremental synchronization. Even if users aren't using new data structures or commands from the new version, downgrades might still fail. For example, in version 7.0, to improve expiration precision, set key value ex ttl is rewritten as set key value pxat timestamp, but some versions like 6.0 do not support the pxat parameter, which could cause downgrades to fail.

[zuiderkwast]

I understand it means extra maintenance burden.

We can discuss how many old versions we want to support. Maybe supporting one previous major version can be enough. To be able to undo an upgrade is a great improvement for critical software.

Regarding generating old RDB formats, it does not need to be identical to how it was done in the past. For example, we don't need to generate ziplist encoding. For sets, we currently use dictIterator to output sets with OBJ_ENCODING_HT, but if we use setTypeIterator instead, thi code it can output any encoding as RDB_TYPE_SET, including the listpack encoded sets, when we want to generate old RDB formats. Then, the code for generating old RDB format will just be a few if conditions.

It's a good finding that also command replication has some special cases. We definitely need cross-version testing...

[madolson]

@valkey-io/core-team Vote about this high level decision.

Support all minors of immediate previous major version. For example, if you are on version 9.0, you can can sync a replica from 8.1 and 8.2, but not 7.2. If the user is using any new features, the replication may get aborted by the replica. Lowest version supported will be 7.2.

[madolson]

Adding this to the 9.0, we will need to make a decision because we will bump the RDB version for Hash field expiration.

[jdork0]

Just adding another vote for this feature to be included. Our telco customer are always insisting on rollback capabilities in our deployments.

[hwware]

Just adding another vote for this feature to be included. Our telco customer are always insisting on rollback capabilities in our deployments.

You mean this vote? https://github.com/valkey-io/valkey/issues/1108#issuecomment-2426925848

[soloestoy]

@zuiderkwast I think we can close this one?

[zuiderkwast]

Relaxed RDB version check solves the problem for RDB-based replication.

However, we didn't make any decision about other things that can prevent downgrade in the future. For example, rewriting commands in replication like you mentioned at some point we started replicating absolute timestamps. Or cluster bus changes. We were discussing earlier to make downgrade possible to the previous major version but we can allow breaking downgrade to older versions, such as more than one year old. A decision like that would be good so we don't need to discuss this every time something comes up.

I hope we can agree to use opt-in or that the primary checks REPLCONF VERSION to make sure it doesn't break old replicas if no new features have been used. We have already started using REPLCONF VERSION like that, for replicating SETSLOT and something similar is done in #1091.

[zuiderkwast]

For reference, this is about etcd downgrade: https://etcd.io/docs/v3.5/downgrades/downgrade_3_5/

TL;DR They say you must first remove any configs and data that is not supported by the downgrade target version.

[zuiderkwast]

What to do with this issue?

In practice, after the introduction of the relaxed RDB version check in #1604, we have been enforcing the following rules for new features:

1. After upgrading, if the user didn't explicitly use any new feature (using a new command or config), we make it possible to undo the upgrade by doing a rolling downgrade back to the version used before the upgrade.
2. If the user has started to use any new feature of the new version, we have no guarantees about what will happen if the user tries to downgrade.

These rules seem to work well. We just need to document this so that users can know what to expect. Can we make a decision about this?

---

Below is a scenario where the relaxed RDB version check is not enough.

So far, the relaxed RDB version check has worked for new commands such as hash-field expiration and it works for atomic slot migration: Don't start using these features until all nodes are upgraded and you are sure you will not go back.

However, following the rules above, we can't make an RDB change like introducing a new encoding and make it enabled by default. For example, introducing listpack for hash-field-expiration (#2618). Consider this scenario, assuming we add this encoding enabled by default in 9.1:

1. A user is running 9.0 and is using hash field expiration.
2. Rolling upgrade to 9.1. The 9.1 nodes encode the hashes with HFE as listpack and writes it in RDB. If this is enabled by default, it happens without any kind of opt-in action by the user. No new commands or config introduced in 9.1 is touched by the user.
3. The user rolls back to 9.0. The 9.0 replica can't understand the HFE encoded as listpack in RDB, even with rdb-version-check relaxed. Downgrade is broken.

This is inconvenient. If we can't add listpack in RDB for hash-field-expiration and enable it by default, then probably users will not enable it. This will affect the RDB size and encoding/decoding speed.

[soloestoy]

What to do with this issue?

In practice, after the introduction of the relaxed RDB version check in #1604, we have been enforcing the following rules for new features:

1. After upgrading, if the user didn't explicitly use any new feature (using a new command or config), we make it possible to undo the upgrade by doing a rolling downgrade back to the version used before the upgrade.
2. If the user has started to use any new feature of the new version, we have no guarantees about what will happen if the user tries to downgrade.

These rules seem to work well. We just need to document this so that users can know what to expect. Can we make a decision about this?

we need two more rules:

1. If RDB encoding changes, users cannot downgrade even if they have not used the new features (we don't want to keep maintaining the old RDB format in the new version).
2. If the mechanism of replication changes, downgrade will not be possible.

[jdork0]

Is there a command that can be used to check if new features are in use that would prohibit downgrade?