oscapd-evaluate should not report that evaluation was done if it wasn't done

[jan-cerny]

I think that following output is horribly confusing, because in this case neither CVE scan nor Standard compliance scan was performed.

[root@thinkpad ~]# oscapd-evaluate scan --no-cve-scan --no-standard-compliance --output output/
INFO:OpenSCAP Daemon one-off evaluator 0.1.7
INFO:Successfully imported 'docker' and 'Atomic.mount', container scanning enabled.
INFO:Evaluated EvaluationSpec, exit_code=0.
INFO:[100.00%] Scanned target 'localhost'

The only thing that has been done was that applicable CPEs were determined, but I will not say that the "target was scanned". I see determining CPEs as a special use-case, that has to be done before scan.

I know that technically it's a scan, because it uses oscap xccdf eval under the hood, but user doesn't have to know that.

[jan-cerny]

It's even more confusing for me when I run oscapd-evaluate scan --no-cve-scan --output output and it says immediately:

INFO:OpenSCAP Daemon one-off evaluator 0.1.7
INFO:Successfully imported 'docker' and 'Atomic.mount', container scanning enabled.
INFO:Evaluated EvaluationSpec, exit_code=0.

and then nothing happens for tens of seconds and my CPU cooler starts to spin. It gives me an impression that it's stalled and I should kill it. But actually it's evaluating "Verify and Correct File Permissions with RPM" rule, which take some time.

If I'm patient enough, I'll get

[root@thinkpad ~]# oscapd-evaluate scan --no-cve-scan  --output output/
INFO:OpenSCAP Daemon one-off evaluator 0.1.7
INFO:Successfully imported 'docker' and 'Atomic.mount', container scanning enabled.
INFO:Evaluated EvaluationSpec, exit_code=0.
WARNING:Evaluated EvaluationSpec, exit_code=2.
INFO:[100.00%] Scanned target 'localhost'

I think that's not a good user experience.

[jan-cerny]

@mpreisler I'm interested in your opinion on this. I suggest handling CPEs determination in a special way so that it is not reported as a scan.

Also I think that we could show the progress while evaluating, that would be beneficial especially for profiles with many rules. Waiting for tens of seconds without any output isn't user friendly.

[jan-cerny]

Any suggestions?

[jan-cerny]

@OpenSCAP/daemon-developers ???

[mpreisler]

@mpreisler I'm interested in your opinion on this. I suggest handling CPEs determination in a special way so that it is not reported as a scan.

Also I think that we could show the progress while evaluating, that would be beneficial especially for profiles with many rules. Waiting for tens of seconds without any output isn't user friendly.

Yeah, sure.