Add a --fail-defined flag to opa exec

[phantlantis]

What is the underlying problem you're trying to solve?

I see that there is an option in opa eval that tells opa to return a nonzero exit code if anything comes back defined in the result[] block of a policy evaluation - referred to as --fail-defined. When attempting to run opa exec with this flag, however, I receive an error that the flag is not recognized. Because of this, when I run opa exec and receive defined results for a deny decision, the CLI returns a zero exit code which creates extra work to orchestrate with tools that are dependent on CLI exit codes for decisions.

Describe the ideal solution

Ideally, implementing the --fail-defined flag in opa exec so that if something comes back defined in a result[] block, then a nonzero exit code is returned via CLI.

Describe a "Good Enough" solution

Some additional documentation examples of how to achieve equivalent functionality in opa eval would also probably be sufficient.

Additional Context

[stale[bot]]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days.

[byronic]

Yeah, this is pretty important. See from the output here that both decisions exit with a 0 code, making this useless for a pipeline or automated run:

$ opa exec --decision terraform/analysis/authz --bundle ./policy plan.json
{
  "result": [
    {
      "path": "plan.json",
      "result": true
    }
  ]
}
$ echo $?
0
$ opa exec --decision terraform/analysis/authz --bundle ./policy plan.json
{
  "result": [
    {
      "path": "plan.json",
      "result": false
    }
  ]
}
$ echo $?
0

Given that it's a feature I need, I'm going to get started on it.

[anderseknert]

Great! I'll assign you to the issue then 👍

[byronic]

Thanks much! I am in-progress on this, currently troubleshooting the build. Hopefully I'll have something to show by the end of the week

[anderseknert]

No stress! If you need any help getting things set up, the #contributors channel in the OPA Slack is a great place to ask questions 😃

[byronic]

My first pass at this is done, but I expect some changes to come up on review. Sample output of the current PR #5191 :

$ opa exec --decision terraform/analysis/authz --fail-defined --bundle ./policy plan.json
{
  "result": [
    {
      "path": "plan.json",
      "result": true
    }
  ]
}

$ echo $?
0

$ opa exec --decision terraform/analysis/authz --fail-defined --bundle ./policy plan.json
{
  "result": [
    {
      "path": "plan.json",
      "result": false
    }
  ]
}
{"err":"exec error: there were 1 failures and 0 errors counted in the results list, and --fail-defined is set","level":"error","msg":"Unexpected error.","time":"2022-09-27T22:31:31-05:00"}

$ echo $?
1

[byronic]

The related PR has been updated to exit non-zero on any defined result (or error), and zero on undefined results w/ no errors.

So the above comment would show nonzero exit codes for both false and true, but as noted in the PR comment you can condense a truthy value into an undefined result if needed at the policy level -- cf. https://github.com/open-policy-agent/opa/pull/5191#issuecomment-1261105329

[byronic]

This is still in-progress, now with a fresh PR #5295.

[EvertonSA]

hi colleagues,

where can I find examples of how to use?

I cant make this work, tried almost everything using terraform blast radious example

$ opa exec --decision terraform/analysis/score --bundle ../policy/ tfplan.json
{
  "result": [
    {
      "path": "tfplan.json",
      "result": 500
    }
  ]
}
$ opa exec --fail-defined --format pretty --decision terraform/analysis/authz --bundle policy/ tfplan.json
{
  "result": [
    {
      "path": "tfplan.json",
      "result": true
    }
  ]
}
{"err":"exec error: there were 1 failures and 0 errors counted in the results list, and --fail-defined is set","level":"error","msg":"Unexpected error.","time":"2023-02-16T09:[39](REDACT):03Z"}

[EvertonSA]

I really wanna use this feature, but for a for a first time OPA user, I have not managed to get this working on the terraform blast example. not even on the internet, have found no examples on how to use exec --fail.

[anderseknert]

Can you try it out with some input that won't render a result? Since you're getting a result back from your call, you're not "failing" and --fail will have no effect.

@srenatus may correct me if I'm wrong, but I believe --fail only fails on undefined — i.e. if terraform/analysis/authz in your example does not evaluate at all. Partial rules are never undefined though 🤔 When using opa eval it is possible to craft the query in such a way that an empty set may be turned into an undefined result. That's probably not the case for opa exec as you provide a path to a decision rather than a query. Perhaps we should add a --fail-empty flag to opa exec to account for this..

[ramsharan-choudhary-bzy]

Hi All,

I am still facing this issue. So far OPA is not mature enough to use it for terraform. Not much help present online and open issues. Did anyone so far use OPA in pipeline. My problem is when I used OPA in GitHub actions even when result array is empty it is exiting with non-zero code in case of --fail-defined.

[anderseknert]

Hi @ramsharan-choudhary-bzy!

Yeah, I think the issue still persists for when the query involves evaluating a set, and getting an empty one back in the result. This can be dealt with more easily in opa eval where you can be more flexible in crafting the query, but definitely seems like something we should improve for opa exec. I have added a feature request issue for this just now: https://github.com/open-policy-agent/opa/issues/6132