Restructure how we mention the JSON Schemas

[karenetheridge]

This is somewhat related to #4831...

Currently the schemas (used to validate OpenAPI documents, as well as provide aids to implementations parsing the OAD by identifying locations of schemas and other identifiable types) are described by https://spec.openapis.org as "non-normative", and by the specification as "non-authoritative".

• "non-normative" means that the schemas are not an official formal part of the specification, but only there to be helpful and aid understanding;
• "non-authoritative" means that the schemas are not from an official source.

I believe neither of these things are true: the schemas are part of the specification in that they capture, to the best of their ability, the constraints and restrictions described by the specification, and we take effort to ensure that they are correct and in sync with the changes to the specification document, and the schemas do come from an official source (they are published from the very same repository as the specification text itself, authored by the same trusted people).

I would like the schema links to be more prominent in the published specification, listed at the top of https://spec.openapis.org/oas/latest.html (perhaps below "Editors"), or farther up as two new subsections: "Latest Schemas" (four bullet points) and "Previous Schemas" (a single link to https://spec.openapis.org/#specifications). This could be templated so as not to require any manual spec editing should we wish to release a schema update/fix. Alternatively, we could make use of this via ReSpec (see https://respec.org/docs/#otherLinks)

The paragraph at https://spec.openapis.org/oas/latest#schema could be rewritten to indicate that the schemas can be trusted to perform validation of an OAD: while the schemas cannot cover every requirement and constraint of the specification, validation is guaranteed to NOT produce false failures -- that is, while the schemas cannot promise to fail validation on every invalid OAD, we can promise that we will never fail validation for a valid OAD.

Also, https://spec.openapis.org can contain similar wording changes (removing mention of them being non-normative): the schemas can be used to aid parsing, and will find many structural errors in the OAD, so it is fully expected and recommended that all tools make use of these.

[ralfhandl]

Fully agree with your points on "non-normative" and "non-authoritative": the schemas are normative and supplement the spec, validating against the schemas is necessary for correct OADs, even if it is not sufficient.

Would you be so kind and open PRs with better phrasing for

• the v3.1-dev branch (to be upported to v3.2-dev), and
• the gh-pages branch, file index.md?

Thanks in advance!

[ralfhandl]

I would like the schema links to be more prominent in the published specification, listed at the top of https://spec.openapis.org/oas/latest.html (perhaps below "Editors"),

We can put them in the respec otherLinks field, so they will appear after "Former editors:".

[handrews]

I'm not taking a position on this, but I want to point out that making the schemas normative means that if they say something that disagrees with the specification, then we have a serious problem that needs fixing.

Making them "non-normative" (a.k.a. "informative") doesn't mean we don't care or that they're not official. It just means that they implement the specification rather than defining the specification. I think @OAI/tsc needs to work through what happens if there is a schema bug and we have declared them normative.

An alternative would be to clarify their status as officially maintained and endorsed without setting up a potential conflict of normative sources. There should only be one normative source for any requirement.

This is true within the spec as well, and we could be better about it, but we have improved recently by taking normative requirement terminology out of things like examples. Actually, the examples in the spec are a great analogue for the schemas. They are not normative, but they are just as official as any other part of the spec. The only difference is that if the spec wording and the examples mismatach, the spec wording is what we prefer. At least if we can figure out what the language was intended to mean.

[handrews]

@ralfhandl

Fully agree with your points on "non-normative" and "non-authoritative": the schemas are normative and supplement the spec, validating against the schemas is necessary for correct OADs, even if it is not sufficient.

My previous comment is basically saying that we can designate them to be "authoritative" in some sense without designating them to be "normative."

@karenetheridge

To me, "authoritative" means "if it is wrong we will fix it because we consider it equally important to the project, and not a secondary resource like learn.openapis.org or tools.openapis.org", while "normative" creates potential conflicts that would be problematic and difficult to resolve.

[ralfhandl]

@handrews https://en.wikipedia.org/wiki/Normativity#Standards_documents cites

The CEN-CENELEC Internal Regulations describe "normative" as applying to a document or element "that provides rules, guidelines or characteristics for activities or their results" which are mandatory.[14]

The validation rules in our schemas seem mandatory to me, even if they are not sufficient. I don't think a JSON/YAML document that fails to validate against our schemas can be a valid OpenAPI document.

(Unless there is an obvious bug in a schema such as disallowing field values that are explicitly allowed by the prose spec, or a misspelled fixed field.)

What am I missing?

[handrews]

@ralfhandl I'm philosophizing at this point, so take it with a grain of salt, but...

I think the main issue is the potential for conflicts. But perhaps we can call them both normative but still say that the spec wins? The important thing is to be able to fix the schemas to conform to the spec when there is an error, rather than needing to "fix" the spec to conform to the schemas.

So in that sense, passing validation against the schemas is not mandatory. With the currently published schema, an OAS 3.1 OAD that correctly uses a relative jsonSchemaDialect field will fail validation (if format validation is enabled and implemented correctly). But it's still valid, and tools are still required to support it. Which means we're required to fix the schema.

And while some tools always do schema validation, not all do, and for some it is optional to allow faster loading times. If we make schema validation mandatory, then we're saying that all conforming tools MUST perform schema validation at parse time. And I'm not sure we want to do that. It would be good to research tooling usage more broadly before deciding on such a thing. And particularly keeping in mind use cases like GitHub's massive OAD, where validation can be quite expensive.

[karenetheridge]

I think you're trying to say something we already have noted: that in cases where the specification and the schema are in conflict, the specification prevails.

But this sort of language is totally unhelpful: " But it's still valid, and tools are still required to support it." Tools won't support it if they have the schemas as an integral part of their functioning, and saying "well you have to support it" doesn't change that reality. The actual reality will be that these tools won't support it until they either patch the schema internally themselves somehow, or a fixed version of the schema is released. Either one of those requires human intervention and won't just happen by magic.

[handrews]

The language in the spec is intended to say that tools are, in fact, not required to use the schemas. Up through and including 3.1.0, there weren't any schemas at publication time, so it was not possible to base an implementation on them.

If implementations want to take a dependency on the schemas, that's fine, and we should better-support doing that through package managers and prompt bug-fixes, like all other dependencies are handled. We should do all of that as part of supporting the implementation community.

But that is not the same thing as declaring the schemas to be normative in the sense that the spec is normative. If you want to do that, you need to explain how you handle conflicts, not just in terms of errors, but in terms of how the schemas allow an endless number of invalid OADs. And always will. There MUST be one source of truth, and that is the spec.

[ralfhandl]

Let's take the one schema bug whose fix hasn't been published yet:

• The spec says the Link Object has a server field whose value is a Server Object.

• The schema says the Link Object has a body field whose value is a Server Object, and doesn't allow a server field.

Which field name would you expect an implementation to support?

We can make the future schemas normative, and we MUST add a disclaimer that

1. schema validation is not sufficient for correctness of an OAD, and
2. the prose spec wins if the schemas contradict it, and as a consequence schema fixes may be "breaking" changes.

Side note: if you want the fix for this bug published, please approve and merge

• #4489

[handrews]

@ralfhandl yeah this is a perfect example of why the schemas are informative. And also shows some challenges here- this is a field that very few people use, so the practical impact was low. But it still means that all prior 3.1 schema versions fail to accept all valid OADs, and shouldn't be used. Hopefully the one being published now (I just merged the PR) accepts all valid OADs, and only has the normal "problem" of also accepting many invalid OADs.

[karenetheridge]

If we make schema validation mandatory, then we're saying that all conforming tools MUST perform schema validation at parse time.

There's a few different things going on here. I think we can say that the OAD MUST pass validation against the spec, without saying that an implementation MUST actually do so. If it's passed validation once, and the schema doesn't change, then we know it will continue to pass, and there's no reason to insist the implementation constantly validate it all the time.

As an example, I have the draft2020-12 json schemas pre-loaded in my implementation and they are never validated against themselves, because I know they are already valid. But the JSON Schema specification can still say that the spec metaschemas MUST pass validation against themselves, and I still haven't violated anything in the spec.

I believe we should say that schemas ARE normative, and that OADs MUST pass validation, while also acknowledging that not every invalid OAD will be caught by the schema.

regarding Ralf's question:

Which field name would you expect an implementation to support?

I would expect them to support what's in the spec, not the schema, because we say that when there is a conflict, the specification always prevails. We can assume that any implementation supporting this section of the OAD does not validate their OADs against the spec, or this discrepancy would have been raised immediately. Bugs happen. That's why we fix them as soon as we're aware of them, to allow implementations to properly support everything in the spec. Just because a mistake exists in the schema doesn't mean that it's not normative.

(I think we can also conclude that Link Objects aren't used by many people in OpenAPI, suggesting that maybe we should look closer at them in Moonwalk and see if there's something to improve there.)

[ralfhandl]

we should say that schemas ARE normative

I'd like to do that in v3.3. Right now I don't trust our schemas enough, despite the tests we added.

[karenetheridge]

Right now I don't trust our schemas enough

Tests cannot be 100% exhaustive, but we can still assert that the schemas are believed to be correct. Wouldn't it be weird to imply "hey these are the schemas, but who knows if they're any good.. shrug" ?

[ralfhandl]

I believe the schemas are now more correct than before we started measuring test coverage.

While increasing the test coverage someone correctly remarked that we do not have an aligned and documented test strategy beyond "hit each line and keyword at least once in some test case".

Which is why I see these steps:

• Discuss and align on a test strategy.
• Document and implement the test strategy.
• Make schemas normative (with "incomplete" and "necessary but not sufficient for correctness of an OAD" disclaimers) in the next release.

I don't want to delay 3.2, so I'd strive for 3.3.

What do you think?

[handrews]

but we can still assert that the schemas are believed to be correct

Publishing them means we believe they are correct. We just say that if there is an error, then the specification wins.

There is no way around this. There can only be one source of truth. In the event of a conflict something has to win. They cannot both be equally authoritative.

[ralfhandl]

@karenetheridge We changed how we mention the JSON Schemas in the spec and on https://spec.openapis.org/oas/.

Which further changes do you see necessary?