Updating fact sources' facts generates ghost facts during operations

[Guil33]

Describe the bug If a fact defined in a Fact Source is updated, operations created using this Fact Source will be able to access the old fact value as well, thus creating multiple links (since several values "exist" for this fact).

I will include a detailed example for clarity after the simple To Reproduce part.

To Reproduce Steps to reproduce the behavior:

1. Create a Fact Source, say "Test source".
2. Set a Fact, say "host.network.ip" with the value "192.168.1.1".
3. Create an operation using that Fact Source, with an Adversary that uses the fact you defined, e.g. with an Ability "Ping IP" whose only action is ping #{host.network.ip}
4. Everything's fine!
5. Update the Fact defined in the Fact Source, e.g. "10.0.0.1" instead of "192.168.1.1".
6. Create a new operation with the same Fact Source and same Adversary.
7. The Ability using that Fact is now duplicated, with both the new value and the old value for host.network.ip (2 links are created).

Expected behavior From the documentation: Fact source: You can attach a source of facts to an operation. This means the operation will start with “pre-knowledge” of the facts, which it can use to fill in variables inside the abilities. A fact source is a collection of facts that you have grouped together. A fact source can be applied to an operation when you start it, which gives the operation facts to fill in variables with.

In the light of these lines, I expect that Caldera:

• DOES use the provided Fact Source to fill in facts in links.
• DOES NOT fill in facts in links when I have not provided a Fact Source, or that are not set in the Fact Source.

This means in that case, that the old Fact value(s) should not exist within the context of the operation created with the updated Fact Source (or anywhere else really). Therefore, there should only be one link generated, using the new Fact value, and we shouldn't see the previous Facts values or Facts being used at all.

Detailed example

(Bug in step 6)

1. Create a Fact Source, set a Fact host.network.ip with value 192.168.1.1
2. Create an Adversary with an Ability that uses the fact host.network.ip
3. Create an Operation using that Adversary and Fact Source. As expected, it uses the Fact defined in the Fact Source.
4. Update and save the value of the Fact in the Fact Source.
5. Create an Operation with the same Adversary and Fact Source. As expected, a Fact host.network.ip with the new value is added to the Operation (random print in Operation._init_source in c_operation.py)
6. Despite what is expected or visible in step 5, 2 values exist for the Fact host.network.ip thus 2 links are created.
7. The Fact Source generated by the Operation does show these 2 Facts.
8. Stop the server, inspect object_store. There is only one value for the Fact, and the correct one.
9. Boot the server, create the same Operation again. The same issue happens.

Bonus tests

• If the Fact value is updated one more time, we now have 3 different values, with 1 "real" and 2 "ghosts".
• If a second Fact value is added (same Fact source), we now have 4 values.
• If the newly added value is deleted, we still have 4 values.
• With the host.network.value removed entirely from the Fact Source, there are no Facts added in _init_source, but the 4 values still exist (even before _init_source is called).
• Just to make sure, with all the parameters equal and a new empty fact source, there is no abnormal behavior, i.e. no link is generated because no Fact exists.

Conclusion

The old value for the Fact remains in memory somewhere, even though I have been unable to locate where.

• It's not in the object_store (checked in Step 8).
• It's not seeded in Operation._init_source (checked in Step 5).
• It remains after a server reboot (checked in Step 8-9).

From the bonus tests, I assume there isn't really any way to get rid of old facts / old fact values, save for deleting the Fact Source and creating a new one, because the Facts remain somehow bound to the specific Fact Source they were created in (tested in Bonus).

I did notice that updating the Fact value in the Fact Source triggers a write operation to a yaml file in data/sources, but this file does not seem to be read in later operations (at least I didn't see it), so I'm not sure what to think of it. However, this seems to be the most promising lead at the moment.

Desktop

• OS: Windows 10
• Browser: Firefox 124.0.1

Server

• Caldera: V5
• Unbuntu Server 22.04

Additional context

• This may or may not have its importance, but it looks like facts are taken (with the atomic planner) in the reverse order they were created: if the Fact had value 1, then updated to 2, then updated to 3, the links are generated with the current value first, then the old ones: 3 -> 2 -> 1
• The "ghost" Facts are added with their origin_type as SEEDED, and the source being the correct yaml file in data/sources, which looks ok when reading it (i.e. only 1 Fact, or none at the end of my tests).

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days

[guillaume-duong-bib]

Update: here are the 4 issues I have identified, summed up:

1. A fact that has been loaded into an operation from a fact source will remain in the fact_store even after being deleted from the fact source. Consequences:

• When starting a new operation with the same fact source, the deleted fact will still be used.
• Unnecessary clutter that may become noticeable over time

2. A fact that has been loaded into an operation from a fact source will be duplicated in the fact_store when its value is updated in the fact source (very similar to the first item). Consequences:

• When starting a new operation with the same fact source, the fact will have both the old and the new value (or more if it is updated several times), and there is no way to stop that apart from creating a new fact source or fiddling with the fact_store manually.
• Unnecessary clutter that may become noticeable over time

3. A fact that has been loaded into an operation from a fact source will remain in the fact_store even when the fact source is deleted. Consequences:

• Unnecessary clutter that may become noticeable over time

4. A fact that has been generated by an operation (from parsers) will remain in the fact_store even after the operation is deleted, seemingly never to be used again. Consequences:

• Unnecessary clutter that may become noticeable over time

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days