Linter

[ksze]

Per @Relequestual's recommendation, I'm creating a ticket about a linter and to collect linting rules that people would want.

When authoring a JSON schema, it is desirable to have a linter to check the strictness, specificity, satisfiability, and descriptiveness of a schema. Maybe Ben envisions the JSON schema workgroup to produce a reference linter implementation?

Anyway, I'll start with a list of linting rules that would be desirable (assuming draft 07):

• Objects
  • Is additionalProperties explicitly set? Is it set to either false or a schema (not just true)?
    • If additionalProperties is defined, is propertyNames also defined?
  • Is required defined? Are all fields listed in required?
  • Is there any contradiction among properties, additionalProperties, and required? E.g. this schema is not satisfiable:

    {
      "type": "object",
      "properties": {},
      "required": "foo",
      "additionalProperties": false
    }
    

    foo is required, but is not defined among properties, yet we also disallow additional properties.
  • If either additionalProperties is not false or some fields are missing from required:
    • Are minProperties and maxProperties also defined?
      • Is maxProperties >= the length of required?
  • Are ($ref or $ref-in-allOf) and "additionalProperties": false present together? Usually that's a sign of the author trying to do poor man's inheritance in JSON schema. The "additonalProperties": false will likely cause an object to not validate against the schema, counter to the user's expectations.
• Numbers and integers
  • Does every number or integer have:
    • multipleOf
    • minimum/exclusiveMinimum and maximum/exclusiveMaximum
      • minimum and exclusiveMinimum should not be defined together
      • maximum and exclusiveMaximum should not be defined together
      • Are the (exclusive) minimum and (exclusive) maximum satisfiable?
        • if minimum and maximum are defined, minimum should be <= maximum;
        • if minimum and exclusiveMaximum are defined, minimum should be < exclusiveMaximum;
        • if exclusiveMinimum and maximum are defined, exclusiveMinimum should be < maximum;
        • if exclusiveMinimum and exclusiveMaximum are defined:
          • if type is "integer": exclusiveMaximum - exclusiveMinimum should be > 1
          • if type is "number": exclusiveMinimum should be < exclusiveMaximum
• Strings
  • Do all strings have a format?
  • Do all strings have minLength and maxLength?
    • Is minLength <= maxLength?
• Arrays
  • Do all arrays have items schemas?
  • Are all arrays either:
    • tuple-validated - i.e. items as a fixed-length array of schemas; or
    • list-validated - i.e. items as a single schema
  • in the case of list validation, are minItems and maxItems defined?
    • Is minItems <= maxItems?
  • Is uniqueItems explicitly set? Either true or false is ok. We just want it to be explicit.
• General descriptiveness
  • Does everything have a title?
  • Does everything have a description?
  • Does everything have a $comment?
  • Does everything have a default?
    • Does the default satisfy the schema?
  • Does everything have examples?
    • Do all examples satisfy the schema?

Ideally, the linter shall be able to generate output that is machine-parsable, with the appropriate JSON pointer so that the author can choose to silence/ignore certain warnings.

I know there are more ways for a schema to become unsatisfiable. E.g. a string could have a regex pattern that can only be 3 characters long, but the author also specifies "minLength": 10. This kind of satisfiability violation is too complex to check. I don't expect any linter to be able to check for that.

[Relequestual]

I'll move this elsewhere once an appropriate place exists! 😅

[FadySalama]

Based on w3c/wot-thing-description#1194 it would also be nice to be able to check this as well:

• General descriptiveness:
  • If a schema contains const, does the const satisfy the schema?
  • If a schema contains enum, does each item in enum satisfy the schema?

[mitar]

Two more:

• If $ref points to a remote resource over HTTP, it should use https schema and not http.
• anyOf, allOf, oneOf with just one option (although this is used often as a workaround in old specs for extending $ref-referenced schema).

[ksze]

* If `$ref` points to a remote resource over HTTP, it should use `https` schema and not `http`.

@mitar Not trying to shoot your idea down, but I find this a bit debatable. If I understand the semantics correctly, the "URL" in the "$ref" isn't really a URL. It's more like an identifier. In the most general sense, you could have a resolver that takes that "URL" and resolves it to a schema, which isn't necessarily retrieved via http or https at all.

[mitar]

I agree. The linting tool should definitely have a way to enable/disable rules to use.

[jaketf]

Lint that all string properties that define a regex pattern are internally consistent and obey this regex pattern in their default / example values.

For example, the default and examples in this simple schema are no good! however, they seem to pass in this validator:

{
	"$schema": "https://json-schema.org/draft/2019-09/schema",
    "properties": {
      "lowerLettersOnly": {
        "pattern": "^[a-z]*$",
        "default": "bad_default01",
        "examples": [
          "imatch",
          "i-do-NOT-match01"
        ]
      }
    }
}

I actually think this should be an invalid schema spec, not just a linting issue, @kze should that be a separate GH issue?

[gregsdennis]

Covering default and examples is a good idea.

I actually think this should be an invalid schema spec

This isn't currently possible with stock JSON Schema because we'd have to reference data (the pattern) and apply schema validations against that data. (It is supported with this vocabulary, though.)

[gregsdennis]

I'll move this elsewhere once an appropriate place exists! 😅 - @Relequestual

A place now exists :smile:. I've added some things from here.

[benjagm]

Is it OK for everyone if we create a Github Project dashboard to co-create and visualize all the rule proposals? I suggest this lifecycle:

• Proposal : Someone suggest a new rule.
• Confirmed : The community agrees in adding it.
• Ready to Implement : The rule has all documentation need to implement it.
• Implemented : The rule has been implemented.
• Released : The rule has been released.
• Retired : The rule has been retired.
• Cancelled : The proposal has been cancelled.

[gregsdennis]

Is it OK for everyone if we create a Github Project dashboard to co-create and visualize all the rule proposals? - @benjagm

We need to do this in the linting repo, https://github.com/json-schema-org/json-schema-linting

[gregsdennis]

Based on w3c/wot-thing-description#1194 it would also be nice to be able to check this as well:

• General descriptiveness:
  • If a schema contains const, does the const satisfy the schema?
  • If a schema contains enum, does each item in enum satisfy the schema?

- @FadySalama

I've commented on that issue.

tl;dr - The rules for const and enum should be that they should be defined in isolation. There's no reason to add more constraints to schemas that contain these keywords because they fully constrain the value.

[gregsdennis]

If $ref points to a remote resource over HTTP, it should use https schema and not http - @mitar

URIs are only identifiers, not locators. Whether http or https is used is inconsequential because implementations (by default) shouldn't be accessing the internet to resolve references. These resources should be pre-registered with the implementation prior to validation.