[FR] Add the ability to load schemas that can be referenced in other schemas

[abravalheri]

--store is broken (I think) as a subschema now references another schema (base.json for poetry’s license). I think this is a good opportunity to add the ability to load --extra schemas that can be referenced and --store will load them and the new api would have a way to provide them.

Originally posted by @henryiii in https://github.com/abravalheri/validate-pyproject/issues/144#issuecomment-1920414806

REF: https://github.com/SchemaStore/schemastore/pull/3519#issuecomment-1920450630

[abravalheri]

Additionally, should we allow missing schemas to be downloaded from the base URL of the value specified via --store?

There is an assumption there: when store is given $id always correspond to a downloadable URL. But I guess that we are already using this assumption.

[henryiii]

This was fixed (for now) in SchemaStore by reverting the change in https://github.com/SchemaStore/schemastore/pull/3568, I think because it broke us and one other tool. I personally think it was not the right thing anyway, as Poetry's license field supports more than just SPDX identifiers; "Proprietary" is even in the docs as the correct entry if you don't have an open license. But I could see it being an oneOf in the future. And some other field could eventually reference base.json or some other subschema. So it would be nice to fix it.

Currently, when --store is given, we download all tool.* ref's as URLs, which is valid for SchemaStore. The problem is that there can't be any ref's in the downloaded schemas, as we don't look them up, and we can't load an "extra" schema that is not directly a tool.* schema.

IMO, three change would be ideal: We should add --extra FILE, which loads a schema to be ref'd, --store FILE should recursively load extra refs, and #144 should have a way for a multi plugin to specify extras.

[abravalheri]

--store FILE should recursively load extra refs

This might be a bit tricky. On a related note, SchemaStore's schema-validation.json does seem to list these schemas using externalSchema.

[abelcheung]

Unfortunately, currently it's not just base.json now that makes validation fail with --store:

> validate-pyproject.exe --store=https://json.schemastore.org/pyproject.json .\pyproject.toml
[ERROR] KeyError: 'https://json.schemastore.org/partial-pdm-dockerize.json'

The include chain is pyproject.json -> partial-pdm.json -> partial-pdm-dockerize.json. And we will see more of this in the future for tools with plugin architecture, in particular tox and its whole slew of plugins. For now I used --tool to work around the problem, but in the future that could become unmanageable without recursive loading of $ref.

[henryiii]

What are you passing to —tool to work around?

[abelcheung]

validate-pyproject.exe --store=https://json.schemastore.org/pyproject.json --tool pdm.dockerize=https://json.schemastore.org/partial-pdm-dockerize.json .\pyproject.toml

Haven't tested but should work similarly on mac or linux too. Local relative path also works for both --tool and --store, but it still needs some time fetching all other partial schemas from net. (I end up setting local proxy to serve files from harddisk instead to speed it up)

[henryiii]

Done in #231 and #229.