design spec_version updates

[jku]

Since a incompatible spec version update seems to get more interest (see #2040), we should start with actually designing how spec version numbers are going to work... Who is responsible for setting the version and what tools exist to help there?

As an example, lets say that TUF spec decides that succinct delegations are an incompatible spec addition -- this is a reasonable decision as adding succinct delegations to a targets metadata makes it invalid for older clients. Let's say, this is spec version 2.0.0.

Assume python-tuf supports spec 2.0.0. Now repositories are going to have many reasonable choices. At least these exist:

1. start using 2.0.0 in all new metadata files (but keep old ones as is even when the are resigned). This is what happens by default with current python-tuf: this leads to bugs as soon as
   • succinct delegations are added to existing metadata
   • new metadata is added that old clients should be able to access
2. start using 2.0.0 in all metadata files whenever they get re-signed. Issues appear when:
   • any metadata that old clients should be able to access is resigned
3. start using 2.0.0 in all metadata that contain succinct delegations but legacy version in all others, even newly created metadata -- this allows old clients to keep accessing any metadata that doesn't contain succinct delegations

The last one is arguably most useful but also quite tricky to implement, at least in python-tuf (likely a lot easier in the repository itself): it might be possible to write methods that scans the metadata structure and decides "what is the lowest spec version that this structure supports": this method could be used by the repository when it's updating expiry,version,etc metadata content to also set spec_version at that time.

[jku]

Hmm, I suppose I actually meant that 1.1.0 (instead of 2.0.0) could be a reasonable spec version number for succinct delegations. I'm assuming that means "new clients compatible with 1.1.0 will understand 1.0.x metadata, but old clients will not understand 1.1.0".

[jku]

Hmm, I suppose I actually meant that 1.1.0 (instead of 2.0.0) could be a reasonable spec version number for succinct delegations. I'm assuming that means "new clients compatible with 1.1.0 will understand 1.0.x metadata, but old clients will not understand 1.1.0".

Based on some comments by Justin, I think the intention is not this, but instead the hard compatibility requirement "1.0.0 clients must be able to understand and use all 1.x.y metadata". This is also reasonable, but means there will be more major version bumps in the future.