jsonschema icon indicating copy to clipboard operation
jsonschema copied to clipboard

Published 20 hours ago • verified publisher icon python-jsonschema

Unresolvable JSON pointer: 'meta' when embedding 2020-12 metaschema

Open rabbitfang opened this issue 3 years ago • 4 comments

jsonschema is unable to resolve the "#meta" references in the 2020-12 meta-schema when that meta-schema is embedded in a schema that also uses the 2020-12 meta-schema:

jsonschema.exceptions.RefResolutionError: Unresolvable JSON pointer: 'meta'

Reproducing

  • Python 3.8.10
  • jsonschema 4.2.1

schema.json:

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "$id": "https://example.com/schema/foo",
    "type": "object",
    "properties": {
        "foo": {
            "type": "string"
        },
        "subschema": {
            "$ref": "https://json-schema.org/draft/2020-12/schema"
        }
    }
}

data.json:

{
    "foo": "bar",
    "subschema": {
        "$schema": "https://json-schema.org/draft/2020-12/schema",
        "$id": "https://example.com/schema/bar",
        "type": "object",
        "properties": {
            "bar": {
                "type": "string"
            }
        }
    }
}

main.py:

import json
import os
import traceback

import jsonschema

SCHEMA_2019 = "https://json-schema.org/draft/2019-09/schema"
SCHEMA_2020 = "https://json-schema.org/draft/2020-12/schema"

def read(name):
    base_path = os.path.join(os.path.dirname(__file__))
    with open(os.path.join(base_path, name), 'r') as f:
        return json.load(f)

def test(name, data, schema):
    try:
        jsonschema.validate(data, schema)
    except jsonschema.RefResolutionError as e:
        print('FAIL', name)
        print(traceback.format_exc())
    else:
        print('SUCCESS', name)

def main():
    schema = read('schema.json')
    data = read('data.json')

    test('All 2020', data, schema)

    schema["$schema"] = SCHEMA_2019
    test('Root 2019', data, schema)

    schema["$schema"] = SCHEMA_2020
    schema["properties"]["subschema"]["$ref"] = SCHEMA_2019
    test('Subschema 2019', data, schema)

    schema["$schema"] = SCHEMA_2020
    test('Both 2019', data, schema)

if __name__ == '__main__':
    main()

Expected output

SUCCESS All 2020
SUCCESS Root 2019
SUCCESS Subschema 2019
SUCCESS Both 2019

Actual output

FAIL All 2020
Traceback (most recent call last):
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/validators.py", line 844, in resolve_fragment
    document = document[part]
KeyError: 'meta'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 17, in test
    jsonschema.validate(data, schema)
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/validators.py", line 965, in validate
    error = exceptions.best_match(validator.iter_errors(instance))
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/exceptions.py", line 354, in best_match
    best = next(errors, None)
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/validators.py", line 224, in iter_errors
    for error in errors:
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/_validators.py", line 332, in properties
    yield from validator.descend(
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/validators.py", line 240, in descend
    for error in self.evolve(schema=schema).iter_errors(instance):
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/validators.py", line 224, in iter_errors
    for error in errors:
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/_validators.py", line 298, in ref
    yield from validator.descend(instance, resolved)
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/validators.py", line 240, in descend
    for error in self.evolve(schema=schema).iter_errors(instance):
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/validators.py", line 224, in iter_errors
    for error in errors:
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/_validators.py", line 362, in allOf
    yield from validator.descend(instance, subschema, schema_path=index)
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/validators.py", line 240, in descend
    for error in self.evolve(schema=schema).iter_errors(instance):
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/validators.py", line 224, in iter_errors
    for error in errors:
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/_validators.py", line 298, in ref
    yield from validator.descend(instance, resolved)
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/validators.py", line 240, in descend
    for error in self.evolve(schema=schema).iter_errors(instance):
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/validators.py", line 224, in iter_errors
    for error in errors:
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/_validators.py", line 332, in properties
    yield from validator.descend(
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/validators.py", line 240, in descend
    for error in self.evolve(schema=schema).iter_errors(instance):
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/validators.py", line 224, in iter_errors
    for error in errors:
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/_validators.py", line 46, in additionalProperties
    yield from validator.descend(instance[extra], aP, path=extra)
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/validators.py", line 240, in descend
    for error in self.evolve(schema=schema).iter_errors(instance):
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/validators.py", line 224, in iter_errors
    for error in errors:
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/_validators.py", line 308, in dynamicRef
    with validator.resolver.resolving(lookup_url) as subschema:
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/validators.py", line 752, in resolving
    url, resolved = self.resolve(ref)
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/validators.py", line 783, in resolve
    subschema = self.resolve_fragment(subschema, fragment)
  File "/home/ubuntu/.local/share/virtualenvs/linker-prototype-uUKgQqax/lib/python3.8/site-packages/jsonschema/validators.py", line 846, in resolve_fragment
    raise exceptions.RefResolutionError(
jsonschema.exceptions.RefResolutionError: Unresolvable JSON pointer: 'meta'

SUCCESS Root 2019
SUCCESS Subschema 2019
SUCCESS Both 2019

Additional notes

  • Changing the meta-schema set in data.json's subschema.$schema does not have an effect on outcome.
  • Also happens on jsonschema 4.0.0

rabbitfang avatar Nov 18 '21 18:11 rabbitfang

A minimal example:

import jsonschema

schema = {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "$id": "https://example.com/schema/foo",
    "properties": {
        "subschema": {"$ref": "https://json-schema.org/draft/2020-12/schema"}
    }
}

data = {
    "subschema": {
        "$schema": "https://json-schema.org/draft/2020-12/schema",
        "properties": {
            "bar": {"type": "string"}
        }
    }
}

jsonschema.validate(data, schema)

rabbitfang avatar Nov 24 '21 13:11 rabbitfang

Could I please get another look at this ticket, @Julian? I added a minimal reproduction at https://github.com/Julian/jsonschema/issues/884#issuecomment-977882583.

rabbitfang avatar Jan 18 '22 19:01 rabbitfang

Thanks -- the minimized example certainly looks good -- the next steps would be a change was just merged fixing dynamicRef support (in that all tests of it in the suite currently pass now). Can you test your example with master? If it passes, great, that means it should be fixed with the next release.

If it doesn't, now we need to figure out where the bug is, add a test for it to the upstream suite, and put in a fix.

I doubt I'll have time to look myself in the next 2 weeks at very least, I've got a ton that's about to hit with the academic semester starting, but I can review a PR if needed.

EDIT: Further minimized:

import jsonschema
jsonschema.validate(
    instance={
        "$schema": "https://json-schema.org/draft/2020-12/schema",
        "properties": {"bar": {}}
    },
    schema={
        "$id": "https://example.com/schema/foo",
        "$ref": "https://json-schema.org/draft/2020-12/schema"
    },
)

Julian avatar Jan 18 '22 19:01 Julian

I'm still getting the same error when running on latest main (6787b213d8b5bdad5af29b7c96aeef7ec715aa02).

rabbitfang avatar Jan 24 '22 18:01 rabbitfang

Hello there!

This, along with many many other $ref-related issues, is now finally being handled in #1049 with the introduction of a new referencing library which is fully compliant and has APIs which I hope are a lot easier to understand and customize.

The next release of jsonschema (v4.18.0) will contain a merged version of that PR, and should be released shortly in beta, and followed quickly by a regular release, assuming no critical issues are reported.

It looks from my testing like indeed this specific example works there! If you still care to, I'd love it if you tried out the beta once it is released, or certainly it'd be hugely helpful to immediately install the branch containing this work (https://github.com/python-jsonschema/jsonschema/tree/referencing) and confirm. You can in the interim find documentation for the change in a preview page here.

I'm going to close this given it indeed seems like it is addressed by #1049, but feel free to follow up with any comments. Sorry for the delay in getting to these, but hopefully this new release will bring lots of benefit!

Julian avatar Feb 23 '23 09:02 Julian