Include IR in compliance test cases

[shomron]

What is the underlying problem you're trying to solve?

Swift OPA includes a test harness for running IR compliance tests, whose purpose is to verify correct/expected evaluation behavior and results.

The test data to drive this harness is derived from test cases that live in OPA here. We use a variant of opa-compliance-test to generate extended test files out-of-band, which include the corresponding IR plans, and host the results here.

These test cases may be broadly useful for building compliance tests for other OPA IR language implementations.

Our goal is to spark a discussion on how centralized test cases can continue to grow to include IR plans and serve as the basis for verifying cross-language IR spec compliance.

Describe the ideal solution

We believe a single source of truth for IR spec compliance could be beneficial to the overall ecosystem and allow for compatible implementations to be built with more confidence. To this effect, having a repository of these test cases, already translated into their corresponding IR, could be consumed by test builds for verification. This could be in the OPA repo, or in a separate repo if size becomes a concern.

Describe a "Good Enough" solution

Good enough could be the status quo, with other languages consuming the test corpus either ahead of time or at build time, performing transformations necessary transformations and generating the corresponding IR plans. However, this comes with complexity tradeoffs as well as risk of diverging from the canonical source of truth.

Additional Context

References:

• Test corpus in opa
• Test corpus in swift-opa

[johanfylling]

Another wrinkle to this to consider when deciding on a solution is cross-version compatibility. The IR generated by OPA for a given Rego module might differ across OPA versions, as changes are introduced to the parser, compiler and planner.

[anderseknert]

No objections from me. I imagine we'd just need to automate the process, so that plans are created automatically for any new tests added.. because I'm pretty sure we'll all forget about it if we don't :P

Tangentially related, but something that could be good to do in the same process, as the YAML docs aren't exactly fun to work with: https://github.com/open-policy-agent/opa/issues/7573

[shomron]

Another wrinkle to this to consider when deciding on a solution is cross-version compatibility. The IR generated by OPA for a given Rego module might differ across OPA versions, as changes are introduced to the parser, compiler and planner.

@johanfylling that's an excellent point and one that requires some thought. It feels like there are several use cases we wish to validate:

1. A parser/compiler/planner frontend emits IR that, given a compliant evaluator, produces expected results. Across versions we could test that the IR is semantically equivalent.
2. An IR evaluator, given inputs, produces expected results - with or without regard to what the source Rego looked like.
3. Integration of the above

[srenatus]

1. A parser/compiler/planner frontend emits IR that, given a compliant evaluator, produces expected results. Across versions we could test that the IR is semantically equivalent.

This would be helpful for planner changes in OPA, too. If we recorded the IR of the yaml tests somewhere, we could run a diff against those and the current planner output to assess if any changes are desired. We seldomly have changes to the IR, but it happens (e.g. https://github.com/open-policy-agent/opa/pull/7589).

I could imagine having a workflow that'll update the IR in the yaml files, if that's where we want to store it -- or curate another set of yaml/json files, like to what you did there. If it's in a different repository, we could also tag the corpus according to the OPA version that was used to generate it -- allowing consumers to explicitly tests their evaluators against a given OPA version. We already tag the OPA sources, so if an automated workflow updated the corpus post-merge, I think we get the same options without an extra repository. An extra repository would have some UX advantages for comparing tags, though, GH doesn't make it easy to compare just a subfolder across tags. (That's a GH problem though. You can do that locally with ease.)

💭 Would the situation look different if the CLI tooling made emitting the IR simpler? Maybe with an opa compile -tplan -ojson, or something like that?

[stale[bot]]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days. Although currently inactive, the issue could still be considered and actively worked on in the future. More details about the use-case this issue attempts to address, the value provided by completing it or possible solutions to resolve it would help to prioritize the issue.

[shomron]

💭 Would the situation look different if the CLI tooling made emitting the IR simpler? Maybe with an opa compile -tplan -ojson, or something like that?

@srenatus this would be very useful regardless of the larger issue here! 🙏🏻

[stale[bot]]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days. Although currently inactive, the issue could still be considered and actively worked on in the future. More details about the use-case this issue attempts to address, the value provided by completing it or possible solutions to resolve it would help to prioritize the issue.