DevSkim
DevSkim copied to clipboard
microsoft
JSONschema support for custom DevSkim rules
Is your feature request related to a problem? Please describe. Yes, the documentation is really lacking and I'm having a very hard time getting things to work. A JSONSchema for custom rules would not only help validate the formatting it would be the ultimate documentation and test (that could be embedded even) and really help feedback for custom rule development.
Describe the solution you'd like A complete JSONSchema description for the Custom Rule format for DevSkim.
Describe alternatives you've considered XST.
Additional context
Here is a start, very incomplete but I still wouldn't know how to use all advanced features of DevSkim (such as ymlpath):
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Generated schema for Root",
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"id": {
"type": "string"
},
"description": {
"type": "string"
},
"recommendation": {
"type": "string"
},
"applies_to": {
"type": "array",
"items": {
"type": "string"
}
},
"tags": {
"type": "array",
"items": {
"type": "string"
}
},
"confidence": {
"type": "string"
},
"severity": {
"type": "string"
},
"rule_info": {
"type": "string"
},
"patterns": {
"type": "array",
"items": {
"type": "object",
"properties": {
"pattern": {
"type": "string"
},
"type": {
"type": "string"
},
"scopes": {
"type": "array",
"items": {
"type": "string"
}
}
},
"required": [
"pattern",
"type",
"scopes"
]
}
},
"fix_its": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"type": {
"type": "string"
},
"replacement": {
"type": "string"
},
"pattern": {
"type": "object",
"properties": {
"pattern": {
"type": "string"
},
"type": {
"type": "string"
},
"scopes": {
"type": "array",
"items": {
"type": "string"
}
}
},
"required": [
"pattern",
"type",
"scopes"
]
}
},
"required": [
"name",
"type",
"replacement",
"pattern"
]
}
},
"must-match": {
"type": "array",
"items": {
"type": "string"
}
},
"must-not-match": {
"type": "array",
"items": {
"type": "string"
}
}
},
"required": [
"name",
"id",
"description",
"recommendation",
"applies_to",
"tags",
"confidence",
"severity",
"rule_info",
"patterns",
"fix_its",
"must-match",
"must-not-match"
]
}
}
Also helpful:
- https://transform.tools/json-to-json-schema
- https://www.jsonschemavalidator.net/
- https://json-schema.org/
Thanks for the suggestion. I think this is a great idea and also appreciate your initial stab at it here.
Thank you. If I have time, I'll try to work some more on it later. If this could be included in some kind of unit test or similar (in CI/CD) then consistency is practically enforced, mistakes directly spotted and all potential use cases de facto documented in the JSONschema specification.
In some TS and GoLang projects with similarities I read a bunch of JSON files (like custom DevSkim rules) and validate them against a strictly defined JSONschema. This way there are no unhandled exceptions anymore. The library handles them and tells exactly what failed and what it expected. For example the library Ajv and https://github.com/xeipuuv/gojsonschema.
After analyzing 145 scripts (mostly samples from this and the ApplicationInspector repo) and locally fixing all these mentioned here: https://github.com/microsoft/ApplicationInspector/issues/595 in my test folder. I came up with the following JSON schema.
I'm not sure it is entirely complete, probably not, but it passes all 145 scripts. I tried to use "additionalProperties": false whereever possible to make sure I caught edgecases that were not defined. I also tried adding a minlength of 1 to each string (ensuring it wont be empty and be omitted if not used rather than being empty. I tried defining maxlengths on all strings but I think it's generous enough. The main things I'm not happy about is that in my opinion recommendations minLength should be 1 and omitted when not used. But I left it at 0 for now because all these files use it empty and I didn't feel like fixing them all for now. And that some enum lists now contain the same value in different cases, I think that should be consistent overall.
./test\cryptography\extended.json: '' should be non-empty
./test\cryptography\weakssl.json: '' should be non-empty
./test\frameworks\build.json: '' should be non-empty
./test\frameworks\c.json: '' should be non-empty
./test\os\dynamic_execution.json: '' should be non-empty
./test\security\api\dangerous_api.json: '' should be non-empty
./test\security\attack_surface\outbound_network.json: '' should be non-empty
./test\security\control_flow\dynamic_execution.json: '' should be non-empty
./test\security\control_flow\format_string.json: '' should be non-empty
./test\security\control_flow\permission_evelation.json: '' should be non-empty
./test\security\cryptography\general.json: '' should be non-empty
./test\security\cryptography\hardcoded_tls.json: '' should be non-empty
./test\security\cryptography\hash_algorithm.json: '' should be non-empty
./test\security\cryptography\protocol.json: '' should be non-empty
./test\security\cryptography\random.json: '' should be non-empty
./test\security\frameworks\dotnet_framework.json: '' should be non-empty
./test\security\hygiene\localhost.json: '' should be non-empty
./test\security\hygiene\todo.json: '' should be non-empty
./test\security\manualreview\dynamiccode.json: '' should be non-empty
./test\security\privacy\secrets.json: '' should be non-empty
Any suggestions are welcome, I like to see this adapted into this project and actively worked on as a ultimate syntax reference for writing DevSkim rules. Perpahs this can be included as a first step to validate when using the verify command in CLI.
Here is the new JSONschema:
{
"$schema": "https://json-schema.org/draft-07/schema#",
"title": "Schema for validating DevSkim rules",
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string",
"maxLength": 128,
"minLength": 1
},
"id": {
"type": "string",
"pattern": "^[A-Za-z0-9_-]+$",
"maxLength": 32,
"minLength": 1
},
"description": {
"type": "string",
"maxLength": 256,
"minLength": 1
},
"recommendation": {
"type": "string",
"maxLength": 256,
"minLength": 0,
"$comment": "TODO: This should be minLength: 1 - https://github.com/microsoft/ApplicationInspector/issues/595"
},
"applies_to": {
"type": "array",
"items": {
"type": "string",
"pattern": "^[a-zA-Z0-9._/ -]*$",
"maxLength": 96,
"minLength": 1
},
"additionalProperties": false
},
"tags": {
"type": "array",
"items": {
"type": "string",
"pattern": "^[\\w.:/-]+$",
"maxLength": 96,
"minLength": 1
},
"additionalProperties": false
},
"confidence": {
"type": "string",
"enum": [
"low",
"medium",
"high",
"unspecified"
]
},
"severity": {
"type": "string",
"enum": [
"low",
"medium",
"high",
"critical",
"moderate",
"important",
"manualreview",
"bestpractice",
"unspecified",
"Critical",
"ManualReview",
"BestPractice"
]
},
"rule_info": {
"type": "string",
"maxLength": 32,
"minLength": 1,
"pattern": "^([a-zA-Z0-9_]+\\.md)$"
},
"patterns": {
"type": "array",
"items": {
"type": "object",
"properties": {
"pattern": {
"type": "string",
"maxLength": 1024,
"minLength": 1
},
"xpaths": {
"type": "array",
"items": {
"type": "string",
"maxLength": 256,
"minLength": 1
}
},
"xpathnamespaces": {
"type": "object",
"properties": {
"default": {
"type": "string",
"maxLength": 128,
"minLength": 1
},
"android": {
"type": "string",
"maxLength": 128,
"minLength": 1
}
},
"additionalProperties": false
},
"jsonpaths": {
"type": "array",
"items": {
"type": "string",
"maxLength": 256,
"minLength": 1
}
},
"ymlpaths": {
"type": "array",
"items": {
"type": "string",
"maxLength": 256,
"minLength": 1
}
},
"type": {
"type": "string",
"enum": ["regex", "RegexWord", "regexword", "string", "substring"]
},
"scopes": {
"type": "array",
"items": {
"type": "string",
"maxLength": 64,
"minLength": 1
}
},
"modifiers": {
"type": "array",
"items": {
"type": "string",
"enum": ["i", "m"],
"maxLength": 1,
"minLength": 1
}
},
"confidence": {
"type": "string",
"enum": ["high", "High", "medium", "low"]
},
"_comment": {
"type": "string",
"maxLength": 128,
"minLength": 1
}
},
"required": ["pattern", "type"],
"additionalProperties": false
},
"additionalProperties": false
},
"fix_its": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string",
"maxLength": 64,
"minLength": 1
},
"type": {
"type": "string",
"enum": ["RegexReplace"]
},
"replacement": {
"type": "string",
"maxLength": 64,
"minLength": 1
},
"pattern": {
"type": "object",
"properties": {
"pattern": {
"type": "string",
"maxLength": 128,
"minLength": 1
},
"type": {
"type": "string",
"enum": ["regex", "substring"]
},
"scopes": {
"type": "array",
"items": {
"type": "string",
"maxLength": 32,
"minLength": 1
}
},
"_comment": {
"type": "string",
"maxLength": 128,
"minLength": 1
},
"modifiers": {
"type": "array",
"items": {
"type": "string",
"enum": ["i", "m"],
"maxLength": 1,
"minLength": 1
}
}
},
"required": ["pattern", "type", "scopes"],
"additionalProperties": false
}
},
"required": ["name", "type", "replacement", "pattern"],
"additionalProperties": false
},
"additionalProperties": false
},
"must-match": {
"type": "array",
"items": {
"type": "string",
"maxLength": 1024,
"minLength": 1
},
"additionalProperties": false
},
"must-not-match": {
"type": "array",
"items": {
"type": "string",
"maxLength": 1024,
"minLength": 1
},
"additionalProperties": false
},
"_comment": {
"type": "string",
"maxLength": 128,
"minLength": 1
},
"does_not_apply_to": {
"type": [
"array"
],
"items": {
"type": "string",
"maxLength": 32,
"minLength": 1
},
"additionalProperties": false
},
"depends_on_tags": {
"type": "array",
"items": {
"type": "string",
"maxLength": 32,
"minLength": 1
},
"additionalProperties": false
},
"applies_to_file_regex": {
"type": "array",
"items": {
"type": "string",
"maxLength": 32,
"minLength": 1
},
"additionalProperties": false
},
"overrides": {
"type": "array",
"items": {
"type": "string",
"maxLength": 32,
"minLength": 1
},
"additionalProperties": false
},
"conditions": {
"type": "array",
"items": {
"type": "object",
"properties": {
"pattern": {
"type": "object",
"properties": {
"pattern": {
"type": "string",
"maxLength": 128,
"minLength": 1
},
"type": {
"type": "string",
"enum": ["regex", "regexword", "string", "substring"]
},
"scopes": {
"type": "array",
"items": {
"type": "string",
"maxLength": 32,
"minLength": 1
}
},
"modifiers": {
"type": "array",
"items": {
"type": "string",
"enum": ["i", "m"],
"maxLength": 1,
"minLength": 1
}
},
"xpaths": {
"type": "array",
"items": {
"type": "string",
"maxLength": 256,
"minLength": 1
}
},
"_comment": {
"type": "string",
"maxLength": 128,
"minLength": 1
}
},
"required": ["type", "scopes"],
"additionalProperties": false
},
"search_in": {
"type": "string",
"maxLength": 128,
"minLength": 1
},
"negate_finding": {
"type": "boolean"
},
"_comment": {
"type": "string",
"maxLength": 128,
"minLength": 1
}
},
"required": ["pattern", "search_in"],
"additionalProperties": false
},
"additionalProperties": false
}
},
"required": [
"patterns",
"tags",
"description",
"name",
"id"
],
"additionalProperties": false
},
"additionalProperties": false
}
Here is a quick script I used to validate all rules I had against the JSONschema. Use it after running pip install jsonschema.
import os
import json
from jsonschema import validate, ValidationError
# Load schema
with open('schema.json') as schema_file:
schema = json.load(schema_file)
# Directory to search for JSON files
root_dir = './test'
# Validate JSON files
errors = []
for dirpath, _, filenames in os.walk(root_dir): # Recursively walk through the test folder
for file in filenames:
if file.endswith('.json'):
file_path = os.path.join(dirpath, file)
try:
with open(file_path) as json_file:
data = json.load(json_file)
validate(instance=data, schema=schema)
except ValidationError as e:
# Append the error with full file path
errors.append(f"{file_path}: {e.message}")
except json.JSONDecodeError as e:
errors.append(f"{file_path}: Invalid JSON - {e.msg}")
# Output results
if errors:
print("\n".join(errors))
else:
print("All files are valid.")
Later as JSONSchema draft 7 allows I'd like to see more $comment values added with actual documentation on what it does and how to use it.
Another pro-tip in VSCode I setup in my user settings.json
"json.schemas": [
{
"fileMatch": ["*.devskim.json"],
"url": "file:///C:\\...\\schema.json"
}
],
I renamed the file extension of my custom devskim rules from .json to .devskim.json. And now I get feedback while writing custom DevSkim rules.
As well as autocomplete:
In fact, if we standardize this schema and encourage the use of .devskim.json more we could submit the schema to https://www.schemastore.org/json/ and have it work out of the box with tools like https://marketplace.visualstudio.com/items?itemName=remcohaszing.schemastore.
Thanks for doing all this @JaneX8! I'll review this second iteration and try to fill in anything that might be missing.
Sorry for dropping this earlier. I'll try to revisit in the next couple weeks. I think what's missing here is the schema validation also needs to be integrated into the rule validator itself. If you'd like to propose a PR that includes the schema definition and integrates the schema check into the rule validation code I'm definitely happy to review/work off that.