spike: check runtime attestation for COS_PERSISTENT and COS_OEM contents

[Itxaka]

Kind of the same as the keylime card but apply it to COS_PERSISTENT/COS_OEM files.

This could be done with keylime by applying a policy that only targets files under PERSISTENT mounts.

BUT those files will still need to be measured (no problem on persistent as its RW)

But seems ok, because you could measure those files offline and generate a policy based on those files (i.e. stylus binary and such) and on updates you will firs need to pre-measure what you are gonna deploy and then deploy it.

Anyway scenario for this:

uki/non-uki: doesnt matter node server and attestation server. We dont care about the attestation but node server is a kairos node. You have a binary under PERSISTENT that you use. You want that binary to be measured continuously and compared against a know good value on the attestation server If that value changes, node should either dont run it or even panic, but it should trigger something. I guess this depends on the attestation framework. If the binary is updated with a good known version it should continue working, so policy should allow that.

Additional context: https://github.com/kairos-io/kairos/pull/2981

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[Itxaka]

Expected cmdline:

ima_appraise=fix ima_canonical_fmt ima_policy=tcb ima_template=ima-ng

notice that the policy can be changed to suit your needs: https://ima-doc.readthedocs.io/en/latest/ima-policy.html#ima-measurement-policies

Expected exclude list:

^/(?!oem/|usr/local/)

This will exclude from the verification everything but /usr/local and /oem It will still generate the policy for EVERYTHING in the system, so you will see all the hashes unfortunately, but the verificator will ignore those paths

To create the policy:

keylime-policy create runtime --ima-measurement --rootfs '/' --ramdisk-dir '/boot/'  --excludelist excludelist.txt --output policy.json

Then we just need to add it

keylime_tenant -c add  -u <uuid> -t <ip> --runtime-policy policy.json

Ideally the policy could be generated inside the rootfs artifact, but until runtime we do not have the proper list of stuff under /oem and /usr/local so its kind of difficult

So running that policy generation on the first boot and adding the node + policy is required.

If you are booting several nodes, ideally you would be able to generate the policy offline, i.e.:

• boot the iso under qemu
• install it
• after first boot, generate the polciy
• store it somewhere

then you should be able to use the same policy everywhere for deploying the other nodes and registerem them via the ip from a different machine

NOTICE:

As /usr/local can have binds to dynamic dirs like containers or CNI and stuff liek that, the exclude list migth be not valid for those as they have files that are going to be changing and will not match after a while, so it probably needs to be even more reduce to only cover things that we know are not expected to change like /usr/local/bin for binaries

to verify that its working:

from the same node:

[root@keylime ~]# keylime_tenant --command cvstatus
INFO:keylime.config:Reading configuration from ['/etc/keylime/logging.conf']
2025-07-04 12:34:50.709 - keylime.config - INFO - Reading configuration from ['/etc/keylime/tenant.conf']
2025-07-04 12:34:50.709 - keylime.config - INFO - Applied configuration snippets from /etc/keylime/tenant.conf.d
2025-07-04 12:34:50.709 - keylime.tenant - INFO - Setting up client TLS...
2025-07-04 12:34:50.709 - keylime.tenant - INFO - Using default client_cert option for tenant
2025-07-04 12:34:50.709 - keylime.tenant - INFO - Using default client_key option for tenant
2025-07-04 12:34:50.709 - keylime.tenant - INFO - No value provided in client_key_password option for tenant, assuming the key is unencrypted
2025-07-04 12:34:50.710 - keylime.tenant - INFO - TLS is enabled.
2025-07-04 12:34:50.711 - keylime.measured_boot - DEBUG - mba.elchecking.elchecker: policy names = ['accept-all', 'reject-all']
2025-07-04 12:34:50.711 - keylime.keylime.cmd_exec - DEBUG - Executing command "tpm2_startup --version"
2025-07-04 12:34:50.714 - keylime.elparsing - DEBUG - mba.elparser.tpm2_tools_elparser: TPM2-TOOLS 5.6 detected.
2025-07-04 12:34:50.714 - keylime.tenant - WARNING - Using default UUID d432fbb3-d2f1-4a97-9ef7-75bd81c00000
2025-07-04 12:34:50.715 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): 127.0.0.1:8881
2025-07-04 12:34:50.720 - urllib3.connectionpool - DEBUG - https://127.0.0.1:8881 "GET /v2.3/agents/d432fbb3-d2f1-4a97-9ef7-75bd81c00000 HTTP/1.1" 200 664
2025-07-04 12:34:50.720 - keylime.tenant - INFO - Agent Info from Verifier (127.0.0.1:8881):
{"d432fbb3-d2f1-4a97-9ef7-75bd81c00000": {"operational_state": "Get Quote", "v": null, "ip": "127.0.0.1", "port": 9002, "tpm_policy": "{\"mask\": \"0x400\"}", "meta_data": "{}", "has_mb_refstate": 0, "has_runtime_policy": 1, "accept_tpm_hash_algs": ["sha512", "sha384", "sha256"], "accept_tpm_encryption_algs": ["ecc", "rsa"], "accept_tpm_signing_algs": ["ecschnorr", "rsassa"], "hash_alg": "sha256", "enc_alg": "rsa", "sign_alg": "rsassa", "verifier_id": "default", "verifier_ip": "127.0.0.1", "verifier_port": 8881, "severity_level": null, "last_event_id": null, "attestation_count": 234, "last_received_quote": 1751625290, "last_successful_attestation": 1751625290}}

If the operation state is on "Get Quote" and the "attestation_count" is > 0, that means that its validating every cycle and not failing.

For example, to trigger a failure, if I modify a file in oem:

File has been measured

[root@keylime tmp]# grep /oem/sensitive.txt /sys/kernel/security/ima/ascii_runtime_measurements
10 a200e0c9f668209c52b1758d86a858b4c58a6642 ima-ng sha256:94ec2aceb11df4cbcf36d6103f9e539f778a0c7b252d16b3311c4c66ac882572 /oem/sensitive.txt

Status is ok

{"d432fbb3-d2f1-4a97-9ef7-75bd81c00000": {"operational_state": "Get Quote", "v": null, "ip": "127.0.0.1", "port": 9002, "tpm_policy": "{\"mask\": \"0x400\"}", "meta_data": "{}", "has_mb_refstate": 0, "has_runtime_policy": 1, "accept_tpm_hash_algs": ["sha512", "sha384", "sha256"], "accept_tpm_encryption_algs": ["ecc", "rsa"], "accept_tpm_signing_algs": ["ecschnorr", "rsassa"], "hash_alg": "sha256", "enc_alg": "rsa", "sign_alg": "rsassa", "verifier_id": "default", "verifier_ip": "127.0.0.1", "verifier_port": 8881, "severity_level": null, "last_event_id": null, "attestation_count": 67, "last_received_quote": 1751628622, "last_successful_attestation": 1751628622}}

I modify it and its read:

[root@keylime tmp]# echo hacker > /oem/sensitive.txt 
[root@keylime tmp]# cat /oem/sensitive.txt 
hacker

then the system is marked as invalid

{"d432fbb3-d2f1-4a97-9ef7-75bd81c00000": {"operational_state": "Invalid Quote", "v": null, "ip": "127.0.0.1", "port": 9002, "tpm_policy": "{\"mask\": \"0x400\"}", "meta_data": "{}", "has_mb_refstate": 0, "has_runtime_policy": 1, "accept_tpm_hash_algs": ["sha512", "sha384", "sha256"], "accept_tpm_encryption_algs": ["ecc", "rsa"], "accept_tpm_signing_algs": ["ecschnorr", "rsassa"], "hash_alg": "sha256", "enc_alg": "rsa", "sign_alg": "rsassa", "verifier_id": "default", "verifier_ip": "127.0.0.1", "verifier_port": 8881, "severity_level": 6, "last_event_id": "ima.validation.ima-ng.runtime_policy_hash", "attestation_count": 85, "last_received_quote": 1751628665, "last_successful_attestation": 1751628663}}

And we can see i n the logs that the file doesnt match anymore

jul 04 13:31:05 keylime keylime_verifier[1426]: 2025-07-04 13:31:05.382 - keylime.ima - WARNING - Hashes for file /oem/sensitive.txt don't match a94de6ff40b515e07196c177fb63325e7bf64b608d6c0d9eb5dff1e1299fbf62 not in ['94ec2aceb11df4cbcf36d6103f9e539f778a0c7b252d16b3311c4c66ac882572']

And we can confirm that with the ima hashes from the kernel, where we see that it has now two different hashes:

[root@keylime tmp]# grep /oem/sensitive.txt /sys/kernel/security/ima/ascii_runtime_measurements
10 a200e0c9f668209c52b1758d86a858b4c58a6642 ima-ng sha256:94ec2aceb11df4cbcf36d6103f9e539f778a0c7b252d16b3311c4c66ac882572 /oem/sensitive.txt
10 f73f67c58b3bb8437812d77e44590491b8b3513f ima-ng sha256:a94de6ff40b515e07196c177fb63325e7bf64b608d6c0d9eb5dff1e1299fbf62 /oem/sensitive.txt

So indeed, this is feasible, we just need to clean the instructions a bit and be explicit on the polic as it can be set for binaries only for example, so your binaries are not tampered with and things liek that but thats generic kernel+ima+keylime stuff, not Kairos specific

[mudler]

let's just document the usage for measuring binaries in the persistent partitions and config files in OEM

[Itxaka]

continuing in https://github.com/kairos-io/kairos/issues/3520