Thin DS: Command Line Interface

[Honny1]

Description:

This PR introduces a new flag for generating thin data streams. The --rule-id flag generates a thin data stream for a single rule. The thin data stream for a single rule generated by the --rule-id flag is reduced from 19 MB to approximately 890 KB. Also, the --thin flag is created to generate thin data streams for each rule in the rules in benchmark relevant to the product. The rules use default variables.

Further reduction of CPE, XCCDF, etc. will be done in future PRs. Also, the documentation of the changes will be created in another PR.

A side effect of this PR is automatic XML formatting using Python3+. So in another PR, the XML formatting step can be removed from cmake.

--rule-id RULE_ID

This flag builds a single Datastream as in the case of thick Datastreams but as a thin Datastream with only one rule.

--thin

This flag builds thin Datastreams for each rule relevant to the benchmark. The components of each rule, such as XCCDF, OVAL, and OCIL, are stored in the build directory in ./build/<PRODUCT>/thin_ds_components as files with prefixes for specific component types and the rule id. The final Thin Datastream is stored in the build directory under ./build/thin_ds/. In this case, you disabled the Datastream build version 1. 2.

Review Hints:

To test the --rule-id flag, you can run this command:

./build_product fedora --rule-id audit_rules_privileged_commands_fusermount3

To test the --thin flag, you can run this script:

#!/bin/bash

./build_product fedora --thin

for filename in ./build/thin_ds/*.xml; do
    echo "$filename"
    oscap xccdf eval  --profile "(all)" --results-arf "arf_results/arf_$(basename -- $filename)" "$filename"
done

The script generates a thin Datastream for each rule and then performs a scan using oscap. This test takes more than an hour to run because there are approximately 1830 rules to process and some rules are very memory intensive.

[openshift-ci[bot]]

Skipping CI for Draft Pull Request. If you want CI signal for your change, please convert it to an actual PR. You can still manually trigger a test run with /test all

[github-actions[bot]]

Start a new ephemeral environment with changes proposed in this pull request:

Fedora Environment

Oracle Linux 8 Environment

[comps]

Seems to work for me - at least oscap xccdf eval consumes it and reports a result.

The speed up in calling oscap is nice, even if the content build time takes a lot longer than a "fat" datastream (when building all rules for a product).

$ time oscap xccdf eval --profile '(all)' --rule xccdf_org.ssgproject.content_rule_umask_for_daemons /usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xml
real    0m1.426s
user    0m1.282s
sys     0m0.135s

$ time oscap xccdf eval --profile '(all)' --rule xccdf_org.ssgproject.content_rule_umask_for_daemons ssg-rhel9-ds_umask_for_daemons.xml
real    0m0.178s
user    0m0.161s
sys     0m0.013s

(Some random rule I saw in the list.)

[Mab879]

I tried build the firefox product with --thin and the build failed. I got similar results with the chromium product.

  File "/home/mburket/Developer/ComplianceAsCode/content/build-scripts/build_xccdf.py", line 150, in <module>
    main()
  File "/home/mburket/Developer/ComplianceAsCode/content/build-scripts/build_xccdf.py", line 146, in main
    link_benchmark(loader, xccdftree, args)
  File "/home/mburket/Developer/ComplianceAsCode/content/build-scripts/build_xccdf.py", line 93, in link_benchmark
    link_oval(xccdftree, checks, args.oval, args.build_ovals_dir)
  File "/home/mburket/Developer/ComplianceAsCode/content/build-scripts/build_xccdf.py", line 75, in link_oval
    oval_linker.link()
  File "/home/mburket/Developer/ComplianceAsCode/content/ssg/build_renumber.py", line 171, in link
    self.oval_document = load_oval_document(parse_file(self.fname))
                                            ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mburket/Developer/ComplianceAsCode/content/ssg/xml.py", line 84, in parse_file
    tree = open_xml(filename)
           ^^^^^^^^^^^^^^^^^^
  File "/home/mburket/Developer/ComplianceAsCode/content/ssg/xml.py", line 77, in open_xml
    return ElementTree.parse(filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/xml/etree/ElementTree.py", line 1203, in parse
    tree.parse(source, parser)
  File "/usr/lib64/python3.12/xml/etree/ElementTree.py", line 557, in parse
    source = open(source, "rb")
             ^^^^^^^^^^^^^^^^^^
TypeError: expected str, bytes or os.Path

[Honny1]

@Mab879 The problem you found should be fixed.

[Mab879]

@Mab879 The problem you found should be fixed.

Confirmed, both the firefox and chromium products build now with -t.

Thanks!

[github-actions[bot]]

:robot: A k8s content image for this PR is available at: ghcr.io/complianceascode/k8scontent:11549

Click here to see how to deploy it

If you alread have Compliance Operator deployed: utils/build_ds_container.py -i ghcr.io/complianceascode/k8scontent:11549

Otherwise deploy the content and operator together by checking out ComplianceAsCode/compliance-operator and: CONTENT_IMAGE=ghcr.io/complianceascode/k8scontent:11549 make deploy-local

[qlty-cloud-legacy[bot]]

Code Climate has analyzed commit 6b4b49fa and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 13.7% (50% is the threshold).

This pull request will bring the total coverage in the repository to 58.1% (-0.1% change).

View more on Code Climate.

[jan-cerny]

/packit retest-failed

[jan-cerny]

/packit retest-failed

[jan-cerny]

/packit retest-failed

[jan-cerny]

/packit retest-failed