[Bug] KeyError: 'external_id' navlayers

[AlaZegnani]

Expected Behavior

I'm having trouble working with navlayers the error occurs when specifying taxii as source source='taxii'

for example working with LayerGeneratorcli or UsageLayerGenerator expected behaviour is getting a .json file as an output

But running the following command:

Actual Behavior

layerGenerator_cli --domain enterprise --source taxii --mapped-to S0065 --output generated_layer.json

i get this error message instead

[taxii2client.v20] [WARNING ] [2022-07-01 16:28:30,170] TAXII Server Response did not include 'Content-Range' header - results could be incomplete.
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:30,171] TAXII Server Response with different amount of objects! Setting per_request=1
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:30,665] TAXII Server Response did not include 'Content-Range' header - results could be incomplete.
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:30,665] TAXII Server Response with different amount of objects! Setting per_request=1
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:31,135] TAXII Server Response did not include 'Content-Range' header - results could be incomplete.
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:31,135] TAXII Server Response with different amount of objects! Setting per_request=1
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:31,668] TAXII Server Response did not include 'Content-Range' header - results could be incomplete.
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:31,668] TAXII Server Response with different amount of objects! Setting per_request=1
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:32,137] TAXII Server Response did not include 'Content-Range' header - results could be incomplete.
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:32,137] TAXII Server Response with different amount of objects! Setting per_request=1
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:32,982] TAXII Server Response did not include 'Content-Range' header - results could be incomplete.
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:32,982] TAXII Server Response with different amount of objects! Setting per_request=1
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:34,222] TAXII Server Response did not include 'Content-Range' header - results could be incomplete.
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:34,222] TAXII Server Response with different amount of objects! Setting per_request=1
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:34,686] TAXII Server Response did not include 'Content-Range' header - results could be incomplete.
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:34,687] TAXII Server Response with different amount of objects! Setting per_request=1
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:35,149] TAXII Server Response did not include 'Content-Range' header - results could be incomplete.
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:35,149] TAXII Server Response with different amount of objects! Setting per_request=1
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:35,650] TAXII Server Response did not include 'Content-Range' header - results could be incomplete.
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:35,650] TAXII Server Response with different amount of objects! Setting per_request=1
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:36,165] TAXII Server Response did not include 'Content-Range' header - results could be incomplete.
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:36,165] TAXII Server Response with different amount of objects! Setting per_request=1
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:36,668] TAXII Server Response did not include 'Content-Range' header - results could be incomplete.
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:36,668] TAXII Server Response with different amount of objects! Setting per_request=1
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:37,956] TAXII Server Response did not include 'Content-Range' header - results could be incomplete.
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:37,956] TAXII Server Response with different amount of objects! Setting per_request=1
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:38,439] TAXII Server Response did not include 'Content-Range' header - results could be incomplete.
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:38,440] TAXII Server Response with different amount of objects! Setting per_request=1
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:38,900] TAXII Server Response did not include 'Content-Range' header - results could be incomplete.
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:38,900] TAXII Server Response with different amount of objects! Setting per_request=1
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:48,018] TAXII Server Response did not include 'Content-Range' header - results could be incomplete.
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:48,034] TAXII Server Response with different amount of objects! Setting per_request=719
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:57,566] TAXII Server Response did not include 'Content-Range' header - results could be incomplete.
[taxii2client.v20] [WARNING ] [2022-07-01 16:28:57,583] TAXII Server Response with different amount of objects! Setting per_request=719
[taxii2client.v20] [WARNING ] [2022-07-01 16:29:05,951] TAXII Server Response did not include 'Content-Range' header - results could be incomplete.
[taxii2client.v20] [WARNING ] [2022-07-01 16:29:05,993] TAXII Server Response with different amount of objects! Setting per_request=719
[taxii2client.v20] [WARNING ] [2022-07-01 16:29:12,649] TAXII Server Response did not include 'Content-Range' header - results could be incomplete.
[taxii2client.v20] [WARNING ] [2022-07-01 16:29:12,664] TAXII Server Response with different amount of objects! Setting per_request=719
Traceback (most recent call last):
  File "C:\Users\Ala\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Ala\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\Ala\Mitre\mitreattack-python\navlayers\Scripts\layerGenerator_cli.exe\__main__.py", line 7, in <module>
  File "C:\Users\Ala\Mitre\mitreattack-python\navlayers\lib\site-packages\mitreattack\navlayers\layerGenerator_cli.py", line 40, in main
    ug = UsageLayerGenerator(source=args.source, domain=args.domain, resource=args.resource)
  File "C:\Users\Ala\Mitre\mitreattack-python\navlayers\lib\site-packages\mitreattack\navlayers\generators\usage_generator.py", line 28, in __init__
    self.matrix_handle = MatrixGen(source, resource)
  File "C:\Users\Ala\Mitre\mitreattack-python\navlayers\lib\site-packages\mitreattack\navlayers\exporters\matrix_gen.py", line 151, in __init__
    self._build_matrix()
  File "C:\Users\Ala\Mitre\mitreattack-python\navlayers\lib\site-packages\mitreattack\navlayers\exporters\matrix_gen.py", line 377, in _build_matrix
    techs, subtechs = self._get_technique_listing(tac.name.lower().replace(' ', '-'), domain)
  File "C:\Users\Ala\Mitre\mitreattack-python\navlayers\lib\site-packages\mitreattack\navlayers\exporters\matrix_gen.py", line 200, in _get_technique_listing
    tid = [t['external_id'] for t in entry['external_references'] if 'attack' in t['source_name']]
  File "C:\Users\Ala\Mitre\mitreattack-python\navlayers\lib\site-packages\mitreattack\navlayers\exporters\matrix_gen.py", line 200, in <listcomp>
    tid = [t['external_id'] for t in entry['external_references'] if 'attack' in t['source_name']]
  File "C:\Users\Ala\Mitre\mitreattack-python\navlayers\lib\site-packages\stix2\base.py", line 251, in __getitem__
    return self._inner[key]
KeyError: 'external_id'

Steps to Reproduce the Problem

1. for me calling taxii as source anywhere in navlayers causes this ( ToExcel, ToSvg matrix_gen ...)

Possible Solution

A workaround could be working with local stix data but this is not an actual solution. It's basically specifying (source='local',resource='stix_file.json').

Plz note that this is my first time working with mitreattack-python so if i'm making a rookie mistake here i would really appreciate your help

[fabian-marquardt]

I can confirm this bug and I think that I found the source of the problem:

Have a look at attack-pattern with id attack-pattern--970a3432-3237-47ad-bcca-7d8cbb217736. As external sources it lists a source named inv_ps_attacks.

This loop will look for the external_id of all external sources of the attack patterns but will only check if attack appears in the source name, which obviously is too unspecific and matches the object stated above. However, this source will not have an external_id, since it is not coming from MITRE ATT&CK and therefore will raise an exception.

A possible solution would be to check specifically if the source name is mitre-attack, but I am not familiar enough to know if this would break things elsewhere. If this is not acceptable, then we should possibly check if external_id is present, but again I do not know if there are unwanted consequences if we do that 😄

If one of the maintainers can provide some guidance then I can prepare a pull request to fix this.

[clemiller]

This issue was addressed in PR #95. Let us know if you still run into any problems.