Custom format procs are not called when a schema uri is not provided

[josephbhunt]

I have a schema definition that did not specify a schema uri such as "$schema" => "http://json-schema.org/draft-04/schema#"

I created a custom format validation according to the example readme file and registered it for all versions using JSON::Validator.register_format_validaton

I wanted to add this custom format for validation against an existing schema. The schema did not have the line "$schema" => "http://json-schema.org/draft-04/schema#". When I ran the validation using JSON::Validator.fully_validate no errors where found. I verified that the custom format was in the list of validators (@@validators). I also verified that the custom format validation was not being called.

Is this the intended behavior? I thought that setting the $schema uri was optional. I spent a lot of time reading through the source code and docs before I simply tried adding that line and it worked.

[RST-J]

No, this is not intended behavior, it actually is not considered behavior. First, the $schema keyword is recommended but not required. Second, json-schema currently uses draft4 as default if no schema was declared. Hence, format validators should be applied, if they have been registered for draft4, which according to your description is the case.

Could you provide me an MWE (minimal working example) of a schema, data and registered format validator which fails to detect the format validation? I'll look into this then.

[gregoryjscott]

I'm experiencing the same issue that @josephbhunt identified. As requested, here is a MWE built from the example in the README. Note that the schema does not specify the $schema.

  format_proc = -> value {
    raise JSON::Schema::CustomFormatError.new("must be 42") unless value == "42"
  }
  JSON::Validator.register_format_validator("the-answer", format_proc)
  schema = {
    "properties" => {
      "a" => {
        "type" => "string",
        "format" => "the-answer",
      }
    }
  }
  errors = JSON::Validator.fully_validate(schema, {"a" => "23"})
  puts errors.length # => 0

It finds no errors. Put the "$schema" => "http://json-schema.org/draft-04/schema#" back in the schema and it will find one error as expected.

[RST-J]

Ok, I guess, then we should simply add that schema key as a default if none is given as json-schema states that it uses draft-4 as default anyways. Or do you have any objections regarding that this will modify the given schema hash?

[aviflax]

I was just bitten by this as well. I find the current behavior non-intuitive. So I agree this should be changed.

As for the solution, I’m not familiar with the implementation of this library, but it seems odd to me to modify the in-memory representation of the schema. It would make more sense to me that if $schema isn’t specified then the validator is operating in draft4 mode by default, as documented, and in that case it looks for custom validators registered for draft4.

(Great lib BTW!)

[RST-J]

Well, laziness ... :smirk: You are right, of course and I think this should actually be as easy as adding the key on initialization. I'll prepare a fix some time soon, probably this evening.

[aviflax]

:+1: :muscle:

[RST-J]

@gregoryjscott I cannot reproduce this. When I run your example with json-schema 2.6.0 and Ruby 2.2.3 and also 2.2.4 the custom format validation is detected as intended. The tests (should) also cover this case.

Can you provide more details about your setup or examine it for what may be the cause? As this also hit others there potentially must be something not too special.

[gregoryjscott]

@RST-J It looks like this problem was fixed between versions v2.5.1 and v2.5.2 (and I agree that v2.6.0 works correctly as well). I was using v2.5.1 back when I commented on this issue in October. I'm pretty sure that https://github.com/ruby-json-schema/json-schema/commit/84012d872b21d9be584aeba16a3e7fae6c0ff11d is the commit that fixed it. (I tried to use git bisect and the test below to track it down but I had to use git bisect in the opposite way from normal since I was trying to find the commit that fixed the bug instead of the commit that introduced the bug, so I'm not 100% confident that I have identified the correct commit.)

require File.expand_path('../test_helper', __FILE__)

class Issue260 < Minitest::Test
  def test_it
    format_proc = -> value {
        raise JSON::Schema::CustomFormatError.new("must be 42") unless value == "42"
      }
      JSON::Validator.register_format_validator("the-answer", format_proc)
      schema = {
        "properties" => {
          "a" => {
            "type" => "string",
            "format" => "the-answer",
          }
        }
      }
      errors = JSON::Validator.fully_validate(schema, {"a" => "23"})
      assert_equal 1, errors.length
  end
end

However, that doesn't explain why @aviflax is seeing the problem now unless @aviflax is using v2.5.1 or older? As far as I'm concerned this issue can be closed.

[aviflax]

I’m using 2.6.0. I’l see if I can reproduce (tomorrow).

[aviflax]

I’m able to reproduce the behavior with 2.6.0. Example here.

[RST-J]

Ah, this fails on my machine, too. Because you explicitly only register the validator for draft4 which I isn't the case in @gregoryjscotts example. You have two options to immediately fix that: Register the custom format validator without specifying a particular version or if you need this then also register for nil or '' which does not need to make sense to you, as this is internal and should work automatically but then you're helped right away.

In the meantime I'll reconsider how this is internally handled and whether it is easy to get rid of that strange '' thing.

[aviflax]

Ah, OK, I see. Thanks for explaining!

The reason I registered the validator for draft4 is because:

1. That’s what the documentation shows.
   1. And the docs don’t mention that you can register a format for any/all versions with nil (which BTW doesn’t seem very intuitive; a constant like ALL would be clearer).
2. It made sense given my mental model of the library; based on the readme I got the impression that draft4 is kinda just always the default mode for everything. So I figured since my schema would be validated using draft4, then registering my format validator for draft4 would work.

So it might be worth maybe tweaking the docs, adding an ALL constant, and/or making this case just work as I thought it would.

Thanks for taking this seriously!

[RST-J]

Hey @aviflax, does the workaround work for you? Because it'll probably take some time, until I find time to fix that - I have to take care of some serious stuff down there in reality. So, if otherwise someone else prepares a PR for this in the meantime, I wouldn't get too mad about it.

And, to prevent another gotcha although I don't think it's very likely: Registering a format validator for nil in the current version is not registering it for all versions. It is only registering it for schemas that have no version specified. Looking back, I don't know why I implemented it this way. Probably because I thought there must be some way to cover this but didn't take it seriously as I thought, one would always specify a version although you may not (such assumptions tend to turn out wrong nearly always). And it just didn't come to my mind, that no version automatically means draft4.

[aviflax]

Yeah, I’m OK for now. Thanks!

[jmaheshkumar]

I am not able to get the custom format error messages, even I have used the "$schema" => "http://json-schema.org/draft-04/schema#". Please give me a solution to get the custom format error message Or How to customise the current error messages.