Allow more options for opa test `--ignore`

[oren-zohar]

Short description

We moved to work with the bundle option on our rego project and had to change the tests to support it, unfortunately, our coverage flow is now broken since the coverage option always yields 100% when working with bundle (https://github.com/open-policy-agent/opa/issues/3324). Instead, we tried to --ignore the tests that will fail when running not in a bundle, but we couldn't get to make it work.

Examples:

opa test compliance --ignore="compliance/lib/output_validations/output_validations.rego"                                                                                                                                                                                                                                     
compliance/lib/output_validations/output_validations.rego:
data.compliance.lib.output_validations.test_validate_rule_metadata: FAIL (59.934417ms)
--------------------------------------------------------------------------------
PASS: 377/378
FAIL: 1/378
(base) 

• OPA version: 0.39.0

[anderseknert]

Thanks for reporting this @oren-zohar 👍

While this pertains to opa test rather than opa build, I think this might be a sympptom of the same issue as reported here: https://github.com/open-policy-agent/opa/issues/2944

Unfortunately I remember it being more involved than it seemed to be at first glance :/

[stale[bot]]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days.

[anderseknert]

I just tried this to see if the other fixes in this space had accidentally fixed this is as well. It had not, but what's interesting is that this works as long as only the filename is provided:

❯ tree
.
├── policy.rego
├── sub
│   └── test2.rego
└── test1.rego

1 directory, 3 files

❯ opa test .
sub/test2.rego:
data.test.test_2: FAIL (176.292µs)
--------------------------------------------------------------------------------
PASS: 1/2
FAIL: 1/2

❯ opa test --ignore sub/test2.rego .
sub/test2.rego:
data.test.test_2: FAIL (130.542µs)
--------------------------------------------------------------------------------
PASS: 1/2
FAIL: 1/2

❯ opa test --ignore test2.rego .
PASS: 1/1

[anderseknert]

So, I've now looked into the code in question, and confirmed the above — the filter is currently based on the name of the file only, and not the path (whether relative or absolute). It is however possible to ignore a directory entirely using only the name of the dir, as that too is considered a "file". So following @oren-zohar's example above, one could do either:

opa test --ignore output_validations.rego .

To ignore any file named "output_violations.rego", while hoping no other file has that same name, or

opa test --ignore output_validations .

To ignore the "output_violations" directory entirely.

Until the --ignore flag is enhanced to accept a path (relative or absolute), the best bet is probably to use a unique filename for those files one wishes to ignore, and provide them to the test command.

[26tanishabanik]

@anderseknert , does #98 fix this issue? If not, where would be the best point to get started?

[26tanishabanik]

I have the directory structure like this:

> tree             
.
├── example.rego
├── example_test.rego
├── first.rego
├── first_test.rego
└── tests
    ├── first.rego
    └── first_test.rego

2 directories, 6 files

And when I changed the function GlobExcludeName(pattern string, minDepth int) in loader.go under opa/loader/, the command gives this output, seems like it is ignoring all the root folder test files too:

// GlobExcludeName excludes files and directories whose names do not match the
// shell style pattern at minDepth or greater.
func GlobExcludeName(pattern string, minDepth int) Filter {
	return func(abspath string, info fs.FileInfo, depth int) bool {
		fileInfo, err := os.Stat(pattern)
		if err != nil {
		   return false
		}
		if fileInfo.IsDir() {
			return true
		}
		match, _ := filepath.Match(pattern, info.Name())
		return match && depth >= minDepth
	}
}

Output:

> ../opa/opa_darwin_arm64 test --ignore tests/ .

I wonder, what am I missing exactly?

[saxenaaman628]

I want to work on this issue @anderseknert

[anderseknert]

You're most welcome to do so @saxenaaman628 👍 I do think fixing this issue might be more challenging than it seems, so don't hesitate to reach out should you need some support. Either here in the issue or use the #contributors channel in the OPA slack, which is pretty good for more frequent back-and-forth conversations.

[saxenaaman628]

I am a little unsure about where should I approach it. @anderseknert What should be the flow?

[anderseknert]

I'd probably try and step through this in the debugger to see where things go wrong, or in other words why the full path isn't considered for the ignore.