zally
zally copied to clipboard
zalando
[EPIC] Rule Sets
This Epic proposes a concept of Rule Sets. This is a grouping mechanism for rule checks and aims to make Zally implementation more general and allow usage beyond Zalando.
As a Zally user I want to be able to package rule checks in groups And I can better group the rules (by topics, technical level, organization, etc.) So that I can easily implement add my own rule sets to a Zally installation without changing source code of the project
Proposal: Rule Sets
Entities
Check
A check implements one aspect of a Rule. A Rule contains one or multiple checks (1-to-n relationship).
Check = (Description, CheckImplementation)
Example: "HTTP method have to correspond with the response code as specified in the standard"
Rule
A rule is an abstract entity, which represents an external guideline which can be automatically checked.
Rule = (Id, Title, Type, ExternalLink, Checks) with Type ϵ {MUST, SHOULD, MAY}
Example: "Use standardized HTTP response codes"
Rule Set
A rule set is a bundle or a package of Rules. It should represent a logical grouping of rules (e.g. one company's guidelines). Rule Sets are developed, tested and published as separate packages. Similar to a plugin mechanism, packages can be added to a Zally installation. Zally server is able to discover them on the startup.
RuleSet = (Id, Rules)
Example: "Zalando RESTful API and Event Scheme Guidelines"
Configuration
All discovered Rule Sets are active per default. Administrator of Zally installation can deactivate Rule Sets (per rule-set-id) and specific Rules (rule-set-id-rule-id) via configuration.
Working Packages
- Ignore Rules by External Ids - #558 ✅
- Core Interfaces - #559 ✅
- Package Discovery - #560
- Statistics for Rule Sets - #561
- Documentation - #562
Preferably, we should work documentation-driven. 1-4 working packages have to be done sequentially.
@netme @maschleg @tkrop @mfellner @roxspring please have a look and give some input :) I'll define some work packages once you approve this proposal.
Generally looks good to me.
Where do violations fit in? Based on the existing architecture I'm presuming that each check can produce 0..1 violation, and therefore going forward a rule can produce many violations. (I'm tempted to suggest that each check should be able to produce many violations with each violation referencing a single location within the swagger, as opposed to the current model where a violation can point to multiple occurrences within the swagger definition).
Does "Zally operator can deactivate ..." imply that just the end user can deactivate thing? Would be good if as the administrator of my installation I can also deactivate so that end users don't need to repeat that configuration.
@roxspring Thanks for your input!
Where do violations fit in?
The interfaces are to-be-defined. I also like your idea. Alternatively, we can do:
- Check returns Check Violations: check-specific description + set of Paths
- Rule returns Rule Violations: rule-specific description + set of Check Violations
Does "Zally operator can deactivate ..." imply that just the end user can deactivate thing?
No, it was bad wording. I meant the administrator. I'll rephrase this.
@maxim-tschumak , thanks for a nice summary, that's exactly what we have discussed before. Let's create tickets out of this epic and start bringing Zally to 1.5
Just wanted to make sure I'm understanding the distinction between the various concepts and trying identify the principles of identifying rules and checks. For example looking at the Zalando Guideline 150:
The RuleSet would be: Zalando Guidelines
The Rule would be the heading from the particular section: MUST Use Specific HTTP Status Codes [150]
The checks would be... one per statement/aspect?:
- You must not invent new HTTP status codes; only use standardized HTTP status codes and consistent with its intended semantics.
- You should use the most specific HTTP status code for your concrete resource request processing status or error situation.
- When using HTTP status codes that are less commonly used and not listed below, you must provide good documentation in the API definition. Note that this suggests that the severity can vary by Check, not simply at the Type level (although maybe the most severe roles up to the Type level??).
Or is the idea that checks would be even more fine grained and broken down by HTTP Method, HTTP Status Code Family, or similar?
@roxspring Yes, you've provided a perfect example of how Rule Set concept supposed to be implemented. Adding more granularity is still possible in the future, but is not part of this change.
I've added working packages to implement Rule Sets. @netme @maschleg @tkrop @mfellner @roxspring please have a look at the working packages
Current implementation which is in progress proposes to store rules as a separate jar file which should be attached as a dependency to Zally project or should be placed into a specific folder. Then using ServiceLoader (probably, because it is a little bit tricky) these rules will be found by Zally and used for validation.
One of the consequences of this solution that when someone wants to change a rule or add new rule a new jar should be placed into specific folder and server should be restarted to pick up changes. Dynamic class reloading can be also implemented, but usually such solutions either error-prone or too complicated to such a small project as Zally.
Java 9 Modules is also not an option because the goals of it are completely different. In short, it is more about performance and optimization but not dynamic loading.
Proposal
Java has a specification JSR-223: Scripting for the JavaTM Platform which allows to execute scripting languages from JVM. There are a lot of implementations including some attempts to use Kotlin inside.
From all options I would go with Groovy because:
- it compiles to the Java byte code.
- Very easy to integrate into application.
- Rules can be added/updated on the fly via UI or via some API endpoint.
- No need to restart Zally when rules are changed.
- has an integration with popular IDE's what allows to write any rules.
- it is a very flexible language.
If implement it like this the rules can be updated easily. User just need to:
- implement/change rule.
- pack them into archive.
- upload archive into Zally.
- profit!
What do you think?
Personally, I've never had a problem with the idea that rules would have to be compiled, packaged and then placed in some directory for Zally to use next time it starts. My initial reaction to dynamically updatable rules is to be concerned about the security risks more than excited about the short dev-cycle. The idea of custom rules being written in another language (or API) from the core ones seems less than ideal to me, but I guess that's easily addressed by moving existing rules over to the new style too.
All that said, momentum on custom rulesets has stalled so it's definitely good to see some fresh ideas and enthusiasm!
@roxspring
Personally, I've never had a problem with the idea that rules would have to be compiled, packaged and then placed in some directory for Zally to use next time it starts.
Cycle might be the same. The difference only that no restarts needed.
My initial reaction to dynamically updatable rules is to be concerned about the security risks more than excited about the short dev-cycle.
Could you please elaborate a bit about security risks?
The idea of custom rules being written in another language (or API) from the core ones seems less than ideal to me, but I guess that's easily addressed by moving existing rules over to the new style too.
In my opinion there should be no "core" rules at all inside Zally. It is just another set of rules which can be used.
#1289 mentioned the problem of proper configuration resolution. Need to take into account for custom rule sets.