Custom abilities do not support requirement definitions using stockpile basic parser

[timbrigham-oc]

Describe the bug Fact requirements are not respected for custom abilities on v 5.0.0.

This is based on the requirement definition, structured after https://caldera.readthedocs.io/en/latest/Requirements.html. The relationship creation process for the demonstration below is using the factory stock "View remote shares".

To Reproduce

1. Create a new adversary.

2. Use an ability such as "View remote shares" to create facts Notably this should have a parser definition similar to the following.

Parser Module = plugins.stockpile.app.parsers.net_view
Source = remote.host.fqdn
Edge= has_share
Target = remote.host.share

3. Create a new custom ability, for example "OC - Test Display".

Requirement Module = plugins.stockpile.app.requirements.basic
Source = remote.host.fqdn
Edge = has_share
Target = remote.host.share

4. Execute an operation using new adversary. The step "OC - Test Display" will not be listed.

Expected behavior The step "OC - Test Display" should be listed, or reasons for the failure included, either in the web interface or the console output.

Additional

I have checked the YML definition and it matches what looks right from the documentation.

cat ./data/abilities/lateral-movement/b56a61cc-6c3c-419a-98ff-8ed9adf4b814.yml
- tactic: lateral-movement
  technique_name: OC - Test Display
  technique_id: T1021.002
  name: OC - Test Display
  description: auto-generated
  executors:
  - name: psh
    platform: windows
    command: write-host "#{remote.host.fqdn}"
    code: null
    language: null
    build_target: null
    payloads: []
    uploads: []
    timeout: 60
    parsers: []
    cleanup: []
    variations: []
    additional_info: {}
  requirements:
  - module: plugins.stockpile.app.requirements.basic
    relationship_match:
    - source: remote.host.fqdn
      edge: has_share
      target: remote.host.share
  privilege: ''
  repeatable: false
  buckets:
  - lateral-movement
  additional_info: {}
  access: {}
  singleton: false
  plugin: ''
  delete_payload: true
  id: b56a61cc-6c3c-419a-98ff-8ed9adf4b814

[timbrigham-oc]

I take that back, the YML created doesn't quite match.

It is (from created yml)

  requirements:
  - module: plugins.stockpile.app.requirements.basic
    relationship_match:
    - source: remote.host.fqdn
      edge: has_share
      target: remote.host.share

vs (from readthedocs.io)

  requirements:
    - plugins.stockpile.app.requirements.basic:
      - source: host.user.name
        edge: has_password
        target: host.user.password

The line "relationship_match:" doesn't exist in the sample.

[guillaume-duong-bib]

1. I think this issue should be named "Inconsistent requirements definition for custom abilities" or something similar, at least that's what I have understood and what my answer is based on.

2. If you check app/service/data_svc.py, line 201 convert_v0_ability_requirements:

async def convert_v0_ability_requirements(self, requirements_data: list):
        """Checks if ability file follows v0 requirement format, otherwise assumes v1 ability formatting."""
        if requirements_data and 'relationship_match' not in requirements_data[0]:
            return await self._load_ability_requirements(requirements_data)
        return await self.load_requirements_from_list(requirements_data)

These V1 and V0 formats seem to refer to the 2 formats you found. Meaning, both formats should be parsed and loaded as Requirements properly. I have not met your issue while using requirements in custom abilities and adversaries.

My guess is that you have another issue with your example that's not related to these different requirements formats.

• Can you provide the parser and requirements configurations as screenshots rather than as code?
• Can you download your operation full report (the one where you encounter your bug), and check or dump the skipped_abilities section?

[timbrigham-oc]

@guillaume-duong-bib I think you are correct that the issue is in the parser for the definitions. I dug into the source code for the basic parser in stockpile, and I think the source and target fields are reversed.

Please note that in these screenshots I don't have a target field defined for the data.requirements.basic parser ingesting later with requirements. I've done this exact same process with that field populated and got the same results.

A "Hello" to create a custom field with basic parser:

A "World" receiver using the stockpile parser:

A "World" receiver using a customized parser:

This customized receiver is a copy of the parser from stockpile with a bunch of logging turned on, and a couple bits of logic flipped. This is experimental, just trying to understand why this was failing. I'm not much of a Python guy so this could be not the best solution. In basic.py, I updated self.is_valid_relationship with the first argument being link.used. In base_requirement.py, I updated the call to _check_target first argument from relationship.target to relationship.source

The "Hello World" adversary:

The operation:

The results:

[timbrigham-oc]

Just because I love going overkill with diagnostic data, here's the console output for my custom parser changes.

This output includes the changes I noted above.

basic.py | self.is_valid_relationship | first argument is link.used base_requirement.py | _check_target | first argument is relationship.source

testing is_valid_relationship

relationship source __dict__ {'_access': <Access.APP: 0>, '_created': datetime.datetime(2024, 5, 13, 12, 51, 54, 214072, tzinfo=datetime.timezone.utc), '_trait': 'hello', 'value': 'this is a hello', 'score': 1, 'source': '8cbd53db-9231-4372-ac11-9fbce0aa05c9', 'origin_type': None, 'links': [], 'relationships': [], 'limit_count': -1, 'collected_by': [], 'technique_id': None}

relationship edge  exists

relationship target __dict__ {'_access': <Access.APP: 0>, '_created': datetime.datetime(2024, 5, 13, 12, 51, 54, 214093, tzinfo=datetime.timezone.utc), '_trait': '', 'value': None, 'score': 1, 'source': None, 'origin_type': None, 'links': [], 'relationships': [], 'limit_count': -1, 'collected_by': [], 'technique_id': None}

testing target in enforcement keys
self.enforcements.keys
dict_keys(['source', 'edge', 'target'])

validating if in used_facts  {'_access': <Access.APP: 0>, '_created': datetime.datetime(2024, 5, 13, 12, 51, 54, 215099, tzinfo=datetime.timezone.utc), '_trait': 'hello', 'value': 'this is a hello', 'score': 1, 'source': '8cbd53db-9231-4372-ac11-9fbce0aa05c9', 'origin_type': <OriginType.LEARNED: 2>, 'links': ['0acc67c0-5cf8-46cd-927a-2614a4f67e9b'], 'relationships': ['hello(this is a hello) : exists'], 'limit_count': -1, 'collected_by': ['bgnlms'], 'technique_id': 'T1003', '_knowledge_id': UUID('fb4ea13f-41cb-4c75-9213-2ae9d11a04e4')}

against relationship.target {'_access': <Access.APP: 0>, '_created': datetime.datetime(2024, 5, 13, 12, 51, 54, 214093, tzinfo=datetime.timezone.utc), '_trait': '', 'value': None, 'score': 1, 'source': None, 'origin_type': None, 'links': [], 'relationships': [], 'limit_count': -1, 'collected_by': [], 'technique_id': None}

what about relationship.source {'_access': <Access.APP: 0>, '_created': datetime.datetime(2024, 5, 13, 12, 51, 54, 214072, tzinfo=datetime.timezone.utc), '_trait': 'hello', 'value': 'this is a hello', 'score': 1, 'source': '8cbd53db-9231-4372-ac11-9fbce0aa05c9', 'origin_type': None, 'links': [], 'relationships': [], 'limit_count': -1, 'collected_by': [], 'technique_id': None}

__check_target
hello  ==  hello
this is a hello  ==  this is a hello
_check_target passes return true
basic true found

[timbrigham-oc]

And the output from my custom parser, with the original logic -

basic.py | self.is_valid_relationship | first argument is [f for f in link.used if f != uf] base_requirement.py | _check_target | first argument is relationship.target

testing is_valid_relationship

relationship source __dict__ {'_access': <Access.APP: 0>, '_created': datetime.datetime(2024, 5, 13, 13, 27, 58, 353876, tzinfo=datetime.timezone.utc), '_trait': 'hello', 'value': 'this is a hello', 'score': 1, 'source': '0c54e291-ff07-475b-8467-8a384fa94e49', 'origin_type': None, 'links': [], 'relationships': [], 'limit_count': -1, 'collected_by': [], 'technique_id': None}

relationship edge  exists

relationship target __dict__ {'_access': <Access.APP: 0>, '_created': datetime.datetime(2024, 5, 13, 13, 27, 58, 353883, tzinfo=datetime.timezone.utc), '_trait': '', 'value': None, 'score': 1, 'source': None, 'origin_type': None, 'links': [], 'relationships': [], 'limit_count': -1, 'collected_by': [], 'technique_id': None}

testing target in enforcement keys
self.enforcements.keys
dict_keys(['source', 'edge', 'target'])

return false
basic false found

The "validating if in used facts" output doesn't show since it isn't getting iterated through. There is no 'fact in used_facts" statement for the loop it's embedded in.. The [f for f in link.used if f != uf] pulls it out of the array.

[guillaume-duong-bib]

TL;DR: you are using the wrong requirement.

Here's what I can say:

• First, I don't think it makes much sense to define a relationship with no target, i.e. either you configure a parser with only a source, or with all three of source, target, edge. Otherwise, your are defining a relationship between a fact, and nothing.
• You identified correctly the part with [f for f in link.used if f != uf], except that's not a bug, but a feature (always wanted to say that). You can see what this requirement was designed for in the doc and the given example of user:password couple. In your case, the default basic parser doesn't accept the fact because you don't use the target in the command... But that's fine, since we don't actually care about an edge or target in your situation.

Although the basic parser is fine, the basic requirement isn't suited to your example. Use existential instead; this should solve your issue - this did in my tests.

[guillaume-duong-bib]

And here's my test: The adversary's first ability parses this is a hello with 3 different configurations.

Then each fact is used in an ability with the corresponding requirement:

With the basic requirement, none of the 3 last abilities work, but that's expected. With the existential requirement, all 3 of them work.

[timbrigham-oc]

@guillaume-duong-bib, many thanks for the detailed assistance! I'm glad I gave you the chance to say something you've always wanted to. :) It would make a lot of sense if I was misunderstanding something. It seemed odd that the parser was full on not working as designed. I will definitely look into the other parsers in more detail.

Can you clarify what you mean by the following a bit?

I don't think it makes much sense to define a relationship with no target, i.e. either you configure a parser with only a source, or with all three of source, target, edge. Otherwise, your are defining a relationship between a fact, and nothing.

It's a pretty standard thing in English at least to have only a subject and a verb without an object so maybe this is confusing me. However one of the stock abilities parsers does exactly this, so it seems supported in Caldera?

Returning to the behavior of the basic parser, I see in the Parsers docs where you have to have the target defined.

The windows lateral movement guide example has does not have a target for plugins.stockpile.app.requirements.basic about half way down. I'm guessing needs a documentation update for clarity. A dedicated page for the stock parsers and their requirements would be great.. I'd love to be involved in that.

To be sure I understand how this parser should be working, I worked through this again with all three fields populated.

If I'm understanding plugins.stockpile.app.parsers.basic correctly, this should result in source "hello" exists and is equal to "this is hello". The edge is the literal string "exists". The output exists and is set to "this is hello". That's my read of the facts created.

If this is the case, I should have all three conditions present and met. In theory shouldn't I be able to use plugins.stockpile.app.requirements.basic to parse the output? This is "OC - World Default Requirement" for reference.

If so, why is "OC - World Default Requirement" skipped and fail to execute in my operation?

[guillaume-duong-bib]

• About the edge/target: you are right, I did not notice that some stock abilities used this kind of one-sided relationships. In that case, it looks more like a way of storing an attribute about a fact than an actual "relationship" though, which seems necessary in that situation as there is no other way to do this. That said however, you definitely do not need to define the edge "exists" in your case, since if the fact exists, well, it exists.
  • Regarding the Windows Lateral Movement, although I understood why there was only an edge defined, I do not see how that would work, since the requirement not_exists is basic reversed, and thus should fail at line 18. I'll definitely test that when I can (EDIT: no, this works and actually makes sense, since not entering line 19 automatically validates the requirement)
  • Also agreed that more explanations about the parsers & requirements would be useful, I took some time to wrap my head around them... And it looks like I'm not done yet!
• And regarding your last question: that one I'm sure of, it's because you use the wrong requirement. The basic requirement needs the target to be used in the command (it's the [f for f in link.used if f != uf] part), so you have 2 solutions to make "OC - World Default Requirement" work:
  • change your command to write-output "#{hello} world, and #{output} world too." (tested and confirmed). This looks confusing, because that's not really a normal use of relationships.
    • To be honest, although the basic parser is useful for quick tests, I don't understand why it's doing what it's doing when an edge and a target are defined (what's the purpose of it). I created a custom parser for specific needs that takes the output and then defines actually useful relationships, e.g. a very simple {"source": {"trait":"user", "value":"lucy"}, "edge": "has_password", "target": {"trait": "password", "value": "horse-battery-staple"} }. So basically, a fact named user with the value lucy, a fact named password with the value horse-battery-staple, and a relationship between the two with has_password as the edge.
  • change the requirement module to existential (tested and confirmed).

[guillaume-duong-bib]

I've noticed something about the basic parser while going through the stock abilities. It is only ever used either:

• with only a source, no edge, no target;
• with a source, edge, and target, but in these cases the relations are used as an attribute/property storing method, e.g.

            - source: directory.sensitive.path
              edge: has_property
              target: has_been_modified

The above is very different from something like

            - source: user
              edge: has_password
              target: password

in the sense that I do not expect the value of has_been_modified to hold much value apart from the fact that it exists. Actually, these facts has_been_modified are only used with the requirement has_property later, which confirms that theory.

The exception is in 90c2efaa-8205-480d-8bb6-61d90dbaf81b, with

            - source: host.file.path
              edge: has_extension
              target: file.sensitive.extension

but in that case, file.sensitive.extension already exists and thus is not replaced with the value of host.file.path because of how the parser works. Therefore, this relation is really meaningful.

So in summary, this parser is not meant to be used the way you did (and I did in my tests), that is to say with a relation between 2 non-existing facts. It's either used:

1. for a single fact with no relation
2. for a single fact and a property (sure, that's a fact code-wise, but it's treated as a property business-wise).
3. for a single fact linked to an already existing fact

This may have been obvious to some, but this is a good discovery for me, now I understand its purpose properly.

[timbrigham-oc]

Thanks @guillaume-duong-bib, appreciate your insight. I didn't make a lot of headway using that lateral movement guide, so I stopped and designed my own variation, hence the questions here about building out relationships.

The "exists" for the edge definitely wasn't the best example, that was a poor choice on my part. Something more like "has_software_xyz_installed" or "is_network_accessible" is more of what I was thinking. Those cases could still be pulled apart further apart and have a true / false value in the object instead of being implied.

I see what you mean about needing to use the target definition in the command block along with the basic definition, using the output variable in "OC - World Default Requirement" has the anticipated results in the flow. I really wasn't expecting there to be smarts in place about the variables being used when checking if they should be available.

write-output "#{hello} world #{output}"

[timbrigham-oc]

I've been working on a set of ability definitions - one for each parser from stockpile. For each parser one, two or three arguments.

The goal was to create an adversary that I could import into my other adversary definitions that would display exactly what conditions would be met and why as I develop my own emulation plans.

Would either that set of definitions or the chart output be useful to post back here? Seems like something that might be worth adding to the readthedocs documentation.

[guillaume-duong-bib]

I can't say I see precisely what your set does, so I guess if you don't mind sharing it, this may be helpful to some.

[timbrigham-oc]

Sure thing.

Hello facts are defined as follows.
Version 1: Hello + none + none Version 2: Hello + two_part + none Version 3: Hello + three_part + output The values set for "hello" uniquely identify which mapping is being used, making it easy to tell them apart in the matrix. "hello one part", "hello two part", "hello three part".

Each of the "World" command uses #{hello} for V1 or V2, or both #{hello} and #{output} for V3. Each of the "World" commands also has three variants defined, looking for the same set of facts as outlined above. There should be three each for Basic, Existential, No Backwards Movement, Not Exists, Req Like, Universal, Paw Provenance, and Reachable.

This chart is a breakdown of how the various combinations of stockpile rules function for one of my test devices.

A ZIP file of the output so it's reproducible. Please note that this expects there to be a copy of the 'requirements' files under data/requirements/.. That's a remnant of my earlier testing, left in place so I could add debugging output without breaking the production copies of these files if needed.

hello world definitions.zip