Backtracking semantics unclear in delegated roles

[jawi]

I'm trying to wrap my head around the whole delegated roles part of the spec. I see the possible use cases for delegating the authority of parts of a repository to different entities outside the main repository authority.

What I'm wondering about is the use of the backtrack flag: as I understand it, it states that "files that are not signed by this role might be provided by other (trusted) roles, go look in other roles as well." As clients should always check all target roles to resolve a particular target file using some sort of priority scheme this flag serves no use.

It might even be harmful: consider that all target files are mapped to only a single target role, with backtrack = false, this could lead to a situation that when another target role (purportedly or intentionally) lists the same target file the client can see different truths depending on the order in which it considers roles. Thus it has to specify a priority scheme and therefore consider all roles in order...

[vladimir-v-diaz]

I wouldn't exactly say that the backtrack flag serves no purpose. Let me try to explain...

By default, the updater will check all target roles to resolve a particular target file. The target roles are visited according to some priority scheme (we support the "ordered delegations" scheme, which is globally applied). However, this behaviour can be overridden by any role with the backtrack flag. When I say that some role can override this default behaviour, it's only for targets requested by the updater for which the role is trusted to specify in its metadata.

Specifically, the backtrack boolean can be set by some role A (in A.json metadata) to indicate to the client's updater (that is requesting a specific target) to continue searching for the target in other delegations if it is not specified in A.json. For example, if backtrack is False and the updater is searching for, but does not find, requests-1.0.tar.gz in A.json, an exception should be raised. Note: I am supposing that A.json is trusted for requests-*.tar.gz targets

On the other hand, if backtrack is True (default) and B.json is also trusted for requests-*.tar.gz targets, the updater should backtrack from A.json and return the target file specified by B.json. That is, sinceA.json came first in the order of delegations, and supposing the client's requests-1.0.tar.gz request is not found in A.jsonand backtrack is True, B.json is visited and the updater finds the client's requested file.

Now, a question one is likely to ask is, "why would A want to prevent the updater from visiting other roles if a requested target file is not found in A.json?" One simple case is to prevent other roles from feeding clients future versions of files that A has not released yet (e.g., preventing B.json from distributing requests-2.0.tar.gz to clients when the latest version is actually requests-1.0.tar.gz). In this case, A wants all requests-*.tar.gz targets available on the repository to be distributed to clients if it's only provided by A.json itself and signed with a secure, offline key.

In a different use case, perhaps a role / project does want some other role (e.g., a role controlled by the repository and signed with an online key) to distribute targets in its name. For community repositories like PyPI, the backtrack flag is very useful (one can even say required for usability and security reasons). On PyPI, this flag allows projects to immediately upload packages and have PyPI sign for them, or later allow projects to claim these packages, sign them with an offline key and prevent clients from trusting packages that PyPI had previously specified. Furthermore, if the PyPI repository were to be compromised (including online keys), client's can only be tricked into downloading projects signed by PyPI. For projects that want to prevent malicious packages in their name to be distributed to clients (following a server compromise), they can sign for packages and utilize the backtrack boolean. Pretty neat!

[vladimir-v-diaz]

I should have also stated that we wish for the framework to be flexible and support many use cases. For example, we'd like to accommodate traditional repositories where all packages are typically signed by repository maintainers, and community repositories where packages can be signed by either external developers or the repository / server. The backtrack flag gives us this flexibility.

If you're still trying to wrap your head around the whole delegated roles part of the specification, perhaps the TUF+PyPI proposals can be of help. See PEP 458 and PEP 480.

For the most part, the TUF specification tells you what properties or security the framework gives a software updater, but not really how to perform delegations on the repository side. The how depends on the software updater that is integrating the framework.

We're considering adding a document to our website that covers more of the how. If you're interested, we can share a paper that goes into more detail on community repositories, priority delegations (the ordered delegations scheme mentioned in the specification), authoritative delegations (delegations that use the backtrack flag), etc.

[jawi]

Thanks for the detailed explanations. I see the value of flexibility and the support of many different use cases, which is why I'm fond of this framework and see many possible uses for it. I probably need to read a little more on the details of community repositories. I'm going to read through the PEPs you provided, and If you can share an additional paper on that, I'd be happy to read that too!

However, I still fail to see how the backtrace flag helps you in preventing requests-2.0.tar.gz from any role other than role A: from an updater's perspective this is an entirely different file than requests-1.0.tar.gz, and not claimed by role A, as it does not provide any indications about the exclusive ownership of the requests-* namespace. Unless targets are actually named requests.tar.gz with a custom tag to differentiate different versions. In that case I see the need for the backtrace flag. I only would suggest to make it not an optional flag, but an explicit one to avoid clients from getting confused over the default value.

[vladimir-v-diaz]

You are correct in saying that A does not indicate, in its metadata, exclusive ownership of some target(s). However, the role that delegates trust of targets to A does decide which target(s) A can sign.

So, due to the way in which roles are visited by the updater and delegations are ordered, it is possible for A to prevent roles that follow it (in the order of delegations) from providing targets that A has been entrusted.

For instance, the top-level targets.json role can delegate trust of files (e.g., files that follow the pattern: requests-*.tar.gz) to three roles: A, B, and C. The order in which these delegations are performed determines how a target requested by a client is searched. If the client asks for requests-1.0.tar.gz, the updater will first consult targets.json. If the target is not found there, it next checks the DELEGATIONS field of targets.json to determine which role to visit next, and which delegations might be trusted with requests-1.0.tar.gz. As it turns out, they are all allowed to sign for this target, because the target fits the pattern requests-*.tar.gz. However, the A delegation is visited first. The A role can decide whether to specify requests-1.0.tar.gz in its metadata, or further delegate trust for it to other roles. If A is allowed to backtrack, and it or its delegations cannot satisfy the requested target, B and C are consulted next.

Here is the format of targets metadata: https://github.com/theupdateframework/tuf/blob/develop/docs/tuf-spec.txt#L680-L687 Note the delegations field on line 686.

Here is the format of the DELEGATIONS field: https://github.com/theupdateframework/tuf/blob/develop/docs/tuf-spec.txt#L711-L723 Note the ROLENAME and the paths it is trusted to sign on line 721.

[vladimir-v-diaz]

Note: I mistakenly said that the backtrack flag can be toggled by A in A.json. It's actually in the role that delegates to A that decides whether A can backtrack or not. This field is not optional, and is always included in the DELEGATIONS field. It defaults to True. For example: https://github.com/theupdateframework/tuf/blob/develop/tests/repository_data/repository/metadata/targets.json#L22

In the linked example, targets/role1 is allowed to backtrack and it is trusted with one path: \file3.txt.

[jawi]

Ah wait, I think I got it now: the delegated roles are consulted for those targets whose path match one of the entries in the delegations.roles.paths list iff the path is not found in targets map of targets.json. In that case, you can narrow the scope of files for which the delegated roles are responsible. And given that a delegated role can delegate parts of its paths to subsequent roles, you can narrow that scope even more.

So, the role that is delegating is responsible for setting the backtrack flag. In that case, I understand the reasoning and use of this flag. Note that the spec does not list the backtrack flag in its example nor describes it, while the formats.py does flag it as optional. Hence my confusion on the optionality of it...

[vladimir-v-diaz]

Ya, you got it! I see why that one mistake about where the backtrack flag is set added to the confusion.

I think we made the backtrack flag optional so that older metadata, that didn't specify it, could be successfully loaded. However, the repository tool that creates TUF metadata does specify the backtrack by default. See https://github.com/theupdateframework/tuf/blob/develop/tuf/repository_tool.py#L1934-L2000

We'll likely not make it optional (in formats.py) in a 1.0 release that includes all of our recent improvements to the framework. The 1.0 release will be backward incompatible.