Add a test for the JSON output of the analyze command

[behnazh-w]

With this PR merged, we have started using Datalog policies in the e2e tests to check the analyze command results. But we no longer check the JSON outputs. We need to add a test for the JSON outputs to catch regressions.

[tromai]

I wonder which test cases do you have in mind for checking the JSON outputs. What I think would work best is to convert the rest of integration test cases (leftover from #764 ). After that, we can go through all of them and pick specific test cases that we wan to check for JSON outputs, then I can update those test cases correspondingly.

[behnazh-w]

I wonder which test cases do you have in mind for checking the JSON outputs. What I think would work best is to convert the rest of integration test cases (leftover from #764 ). After that, we can go through all of them and pick specific test cases that we wan to check for JSON outputs, then I can update those test cases correspondingly.

Yes, sounds good. After converting the rest of the test cases, checking the JSON output for one of the test cases should be enough.

[tromai]

@behnazh-w As #782 is merged, I am thinking of using this test case https://github.com/oracle/macaron/blob/429ba95abc6ee0b917d2115598838b07a4244caf/tests/integration/cases/micronaut-projects_micronaut-core/test.yaml for the JSON report checking. What do you think about it?

[behnazh-w]

Let's use https://github.com/oracle/macaron/tree/429ba95abc6ee0b917d2115598838b07a4244caf/tests/integration/cases/micronaut-projects_micronaut-test for the JSON report as it takes less time in CI.

[tromai]

I can definitely use the test case you suggested. I believe the time taken won't be an issue because we are only adding an extra step in test.yaml to perform the JSON files comparison. Both tests will still be run during the CI :thinking:

[behnazh-w]

I can definitely use the test case you suggested. I believe the time taken won't be an issue because we are only adding an extra step in test.yaml to perform the JSON files comparison. Both tests will still be run during the CI 🤔

I see, in case you add an extra step to the existing test run, micronaut-core is also fine. We should also make sure the JSON output comparison does not check the list of checks, but focuses on the structure. For example, if a new check is added, we shouldn't need to change the expected JSON output.