Provide a helper to fully resolve $refs in a schema

[somiandras]

trafficstars

I want to generate some documentation/description (probably some input forms) from a json schema that contains $refs. Is there a way in jsonschema to retrieve a json schema with $refs resolved and inlined, so I can iterate through the schema?

So from a schema like this:

{
  "definitions": {
    "some_object": {
      "type": "object",
      "title": "My definition",
      "description": "Some description",
      "properties": {
        "name": {
          "type": "string"
        },
        "value": {
          "type": "number"
        }
      }
    }
  },
  "properties": {
    "list_of_objects": {
      "type": "array",
      "title": "Some stuff",
      "description": "List of stuff",
      "items": {"$ref": "#/definitions/some_object"}
    }
  }
}

I want to have this structure in my Python object (notice that the {$ref:...} section is replaced by the actual definition):

{
  "definitions": {
    "some_object": {
      "type": "object",
      "title": "My definition",
      "description": "Some description",
      "properties": {
        "name": {
          "type": "string"
        },
        "value": {
          "type": "number"
        }
      }
    }
  },
  "properties": {
    "list_of_objects": {
      "type": "array",
      "title": "Some stuff",
      "description": "List of stuff",
      "items": {
        "type": "object",
        "title": "My definition",
        "description": "Some description",
        "properties": {
          "name": {
            "type": "string"
          },
          "value": {
            "type": "number"
          }
        }
      }
    }
  }
}

I am aware that with self-referencing schemas this will lead to an infinite structure, but hopefully this is feasible with reasonable limitations (eg. excluding self-refs).

[calvin]

jsonref https://pypi.org/project/jsonref/ resolves references so that you can have fully expanded schemas.

[jpmckinney]

I think @Julian previously decided that this belongs in a separate library: https://github.com/Julian/jsonschema/pull/419

I would like a helper function in jsonschema, though, as https://github.com/gazpachoking/jsonref seems to not be actively maintained (all its tests are failing on CI, so it's not clear that it would even be easy to take maintenance of).

[Julian]

@jpmckinney that was my stance early on, but given nothing popped up, and folks still seem to want this, I'm willing to entertain a helper function (essentially for the reason you mentioned).

So yeah a PR is welcome to add this.

[jpmckinney]

Would python-jsonschema/jsonschema#419 be a starting point? e.g. if you re-open it, it'd be possible to see how bad the conflicts are (if any).

[Julian]

Probably it's a good one yeah -- though I can't reopen it because GH prevents reopening PRs that targeted the old master branch now that it targets main :( -- but if you or someone wants to take what's there and (while preserving authorship of the commits) push it as a new PR it certainly may be a good basis to start from.

[jpmckinney]

Is it possible to edit the PR to point it to a new base branch?

I think I commented here, when I ran into trouble with jsonref and PyPy https://github.com/open-contracting/ocdskit/issues/178, and thought to check whether jsonschema could resolve $ref.

At any rate, the library I was working on has other issues with PyPy. So, I'm not sure I still have a need at the moment.

[Julian]

Is it possible to edit the PR to point it to a new base branch?

Not as far as I know :(

At any rate, the library I was working on has other issues with PyPy. So, I'm not sure I still have a need at the moment.

Fair enough.

[lucasrodes]

I think this would be very valuable!

To load a schema in path that contains references, we are using jsonref like this:

from pathlib import Path
from urllib.parse import urljoin

path = Path(path)

# pathlib does not append trailing slashes, but jsonref needs that.
base_dir_url = path.parent.absolute().as_uri() + "/"
base_file_url = urljoin(base_dir_url, path.name)
with path.open("r") as f:
    return jsonref.loads(f.read(), base_uri=base_file_url, lazy_load=False)

It works fine, but it has some limitations: see https://github.com/gazpachoking/jsonref/issues/60

[Julian]

The thing is, this is actually both impossible to do in general (when considering $dynamicRef) and also explicitly now discouraged by the JSON Schema specification. It's true this has been here for quite awhile and is a lot closer to being implementable now, but can you (or anyone else still interested) perhaps elaborate a bit on what use you have for this functionality?

[lucasrodes]

Hi @Julian, Thanks for the rapid response.

In our case, we are using this functionality for the same use case as the OP. We are centralising the documentation about some fields in a schema. This helps maintenance a lot, since we are surfacing field descriptions and other things in multiple places:

• On hover in VS Code
• On a simple web app we have to introduce some fields in a form-like style.
• And in the future, on our official documentation.

I am still working on it, and the code is quite complex, but I am loading the schema in this script, and later using it to render the help, title and other aspects of the form's input fields.

We happen to use #refs in our JSON schema files, to also simplify maintenance of some field definitions (they might be referenced in multiple places).

[Julian]

Thanks, that helps a little, but I still don't fully follow (I don't think OP shared their own use case originally, they simply asked for the feature). Just to poke further, are the tools you're using otherwise broken with regards to $ref? Is that why you want to inline them? Or otherwise I don't know that I follow what you mean when you say

We are centralising the documentation about some fields in a schema.

Is the point that the tool you're using for form generation doesn't properly support following references? And/or VSCode?

[lucasrodes]

Thanks again. I kind of understood that OP was thinking of maybe creating a form based on the schema values because of them saying:

I want to generate some documentation/description (probably some input forms) from a json schema that contains $refs. Is there a way in jsonschema to retrieve a json schema with $refs resolved and inlined, so I can iterate through the schema?

I could be wrong though.

On our end, the tool that we are using to create the form (streamlit) is just a framework to create an arbitrary form. Then it is up to us to define the (i) number of fields in the form, (ii) titles of the input boxes, etc.

We want to create as many fields in the form as the schema suggests there are. Hence we rely on the schema to create these input boxes in the form. We want to access the descriptions from the fields in the schema, to be able to label the input boxes accordingly. Since some fields in the schema use #refs we needed a way to load the schema in a way that it "replaces references" with their actual values. Like, it basically translates the JSON with refs with a plain JSON without references. That's where jsonref enters the scene.

So far so good with jsonref, except for this tiny issue I recently raised: https://github.com/gazpachoking/jsonref/issues/60

Not urgent, just suggesting that this feature from jsonref could perhaps be supported by jsonschema? Maybe this is out of scope though.

Thanks once again.