Allow registries to reject non-existent subjects in manifests

[dmcgowan]

During conformance testing it was found that registries which require strong references between manifests and blobs fail conformance due to MUST language in the spec requiring acceptance of a manifest referencing a non-existent subject manifest. While subject fields may be described as a weak reference, listing and querying them at large scale may require a strong reference (such as foreign key in a database) or may simply be inheriting the data model used in 1.0 which always had referenced objects (as viewed from the merkle DAG) uploaded first.

The arguments for MUST language was to (1) support registries which may have reference only repositories, storing content elsewhere, and (2) ensure referrers exist at manifest pull time since there is no atomic way to upload referrers with manifests.

For (1) the burden will be on the client to handle this case on upload, as a registry is not required to support such repositories.

For (2) clients can retry or check for freshness when validation is a requirement or clients can ensure tags are only updated once all content is available. Similar issues have occurred in the past with multi-platform images. If images were uploaded before all platforms were available, then clients could see a race condition between the platform they need being built and the image they pull having that platform available. The same solution could apply here, use push by digest or a temporary tag when pushing manifests that should not be considered fully available and "tag" it once complete (via upload of the manifest using tag reference).

Changing the language the MUST to MAY makes most sense here. Additionally we can add guidance in the spec on how to perform manifest uploads more transactionally. In the future we could consider a more explicit way to create and manage transactions.

Related to https://github.com/opencontainers/distribution-spec/issues/340 https://github.com/opencontainers/distribution-spec/pull/341