Consolidate information on loading schemas

[handrews]

It occurs to me that how a schema is loaded and processed is a very fundamental thing that deserves more emphasis than we are giving it.

Loading a referenced schema is an important use case, but generally loading schemas is the same whether you found the schema directly, through $ref, or through hypermedia. As @awwright noted in #808, $ref is not really a special case for this. Additionally, encountering a separate schema resource as a subschema is really another case of loading schemas. We should focus around the resource concept more than the document concept.

Questions around this have been common. The question of indicating a schema from within an instance somehow is one. The question of setting a base URI for a schema read from a filesystem or passed in with no context is another. And then there are questions around $refs among different schema versions (#808) which is related to the question of how to treat embedded schema resources in a single document.

I think we should have a section about loading schemas that covers the general load-and-process model, and notes any specifics needed for $ref, embedded resources, hypermedia, or other cases. My initial thought was to put this early in the spec (general considerations?) but you really need to understand the core vocabulary to do all of the steps.

So I say we should add a new top-level section between section 8 (Core Vocabulary) and section 9 (Applicator Vocabulary) on schema loading. It should include most or all of:

8.2.4.5 Loading a referenced schema (but just Loading a schema) 8.2.4.6 Dereferencing (maybe tweak to cover other ways you get a schema other than just $ref) 8.2.4.3 Guarding against infinite recursion 8.2.4.4 References to possible non-schemas 8.1.4 Detecting a meta-schema (new section) Connecting Instances to Schemas 11 Usage in hypermedia (as subsection) (new subsection) Usage outside of hypermedia (needs better title)

While this sounds like a lot, it's mostly just moving stuff around.

I've gotten several questions that basically boil down to "How do I load schemas?" Consolidating this information would make an essential process much more clear and emphasize its importance. It would also trim the extremely long Core Vocabulary section and avoid some of the really deep nesting that causes key sections to not show up in the Table of Contents right now.

[Relequestual]

This all sounds reasonable. What sort of information do you want to convey in section "Usage outside of hypermedia"?

[handrews]

@Relequestual here's the sort of question that I hope people won't need to ask as much: https://github.com/RicoSuter/NJsonSchema/issues/1090

The answer here is "put appropriate $ids in your schemas, use $refs that match the $ids, don't worry about where you got them from, and the implementation should read the $ids and sort it out for you.

You can figure all of that out from the current draft, but it's not very clear and it's not all in one place.

[Relequestual]

Right. Yes, this often what I try to convey. Might this benefit from being broken out into a few smaller issues?

[handrews]

@Relequestual I'm definitely going to break it into numerous commits. If we can get the ones for just moving blocks of text around merged quickly I can do it as multiple PRs. If everything needs to be up for 2 weeks, I'd rather do one multi-commit PR as OAS 3.1rc1 is supposed to be within the next month.

There's not actually that much new writing to do here, which is why I'm reluctant to file a bunch of issues for things that are essentially "move section X paragraph 2 to section Y paragraph 3." But I'm open to suggestion on how to structure the issues/prs/commits.

[Relequestual]

Given my discovery of git diff blah --color-moved --color-moved-ws=allow-indentation-change I'm fine with multi-commit PR for this.

[handrews]

From doing work on this and the principles issue, I'm also going to create an "Evaluating Keywords" subsection in the "Keyword Behavior" section. Several aspects of loading/processing schemas are about how individual keywords are processed, and being clear about that will make it easier for people to develop useful extension keywords (by not going outside of the processing model).

[Relequestual]

@handrews You mentioned you'd like to consider bumping this issue.

Is there anything you're intended to write which would include requirement keywords (RFC 2119)?

If not, or nothing comes to mind, this issue can be punted to the site repo, and be "information for implementers"

[handrews]

Yeah, I need to go back over this and see what's really essential. I think the work that was done to just organize the sections better (#857 and #860) was helpful, there's a bit of wording clean-up to make the new sections flow better. I'm feeling more able to tackle that sort of thing now, I will try to sort out what still needs to be done and what can be punted in the next few days.

[Relequestual]

OK to close https://github.com/json-schema-org/json-schema-spec/issues/817 and https://github.com/json-schema-org/json-schema-spec/issues/808 and https://github.com/json-schema-org/json-schema-spec/issues/832 in favour of this issue?

[handrews]

@Relequestual yeah that should be fine.

[handrews]

Note that both PRs #914 and #930 represent significant progress towards resolving this issue. The remaining work will be done once those PRs are resolved.

[Relequestual]

Migrating to next draft as this isn't essential for draft 2020 publication.