misp-galaxy MITRE ATT&CK mapping issues with current MISP-galaxy implementation

I am currently working on a new version of the MITRE ATT&CK to MISP-galaxy convertor. (which should be in one script and should also suppor the relationships natively)

The issue I'm encountering is with the enterprise-attack, pre-attack and mobile-attack common entities. They are included in each "domain/phase", but are referred by the same uuid. (as they are the same object)

For example uuid bef4c620-0787-42a8-a96d-b7eb6e85917c. In the MITRE ATT&CK they are used in different bundles. (see below where count > 2)

~/Documents/Projects/MITRE-ATTACK$ fgrep -r -h '"id"' .  | fgrep -v bundle  | sed -E 's/\s+//' | sort | uniq -c  | sort -n | tail -n 14
      2 "id": "x-mitre-tactic--d90bd741-2edb-4e74-8a6f-435143ad7bbb",
      2 "id": "x-mitre-tactic--e78d7d60-41b5-49b7-b0a9-5c5d4cbabe17",
      2 "id": "x-mitre-tactic--f30c2753-e6b2-4186-818d-99b8b1a0322b",
      2 "id": "x-mitre-tactic--f72804c5-f15a-449e-a5da-2eecd181f813",
      3 "id": "tool--9de2308e-7bed-43a3-8e58-f194b3586700",
      3 "id": "x-mitre-tactic--0a93fd8e-4a83-4c15-8203-db290e5f2ac6",
      4 "id": "intrusion-set--090242d7-73fc-4738-af68-20162f7a5aae",
      4 "id": "intrusion-set--23b6a0f5-fa95-46f9-a6f3-4549c5e45ec8",
      4 "id": "intrusion-set--6a2e693f-24e5-451a-9f88-b36a108e5662",
      4 "id": "intrusion-set--8f5e8dc7-739d-4f5e-a8a1-a66e004d7063",
      4 "id": "intrusion-set--d6e88e18-81e8-4709-82d8-973095da1e70",
      6 "id": "identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5",
      6 "id": "intrusion-set--bef4c620-0787-42a8-a96d-b7eb6e85917c",
      9 "id": "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168",

However MISP seems to have included this same object, split over different 'clusters':

~/Documents/Projects/misp-galaxy/clusters$ fgrep -R bef4c620-0787-42a8-a96d-b7eb6e85917c . 
./mitre-enterprise-attack-intrusion-set.json:      "uuid": "bef4c620-0787-42a8-a96d-b7eb6e85917c",
./mitre-intrusion-set.json:        "uuid": "bef4c620-0787-42a8-a96d-b7eb6e85917c"
./microsoft-activity-group.json:          "dest-uuid": "bef4c620-0787-42a8-a96d-b7eb6e85917c",
./mitre-mobile-attack-intrusion-set.json:      "uuid": "bef4c620-0787-42a8-a96d-b7eb6e85917c",
./mitre-enterprise-attack-intrusion-set.new.json:            "uuid": "bef4c620-0787-42a8-a96d-b7eb6e85917c"
./threat-actor.json:          "dest-uuid": "bef4c620-0787-42a8-a96d-b7eb6e85917c",
./mitre-pre-attack-intrusion-set.json:      "uuid": "bef4c620-0787-42a8-a96d-b7eb6e85917c",

This gives the impression that these objects are not identical, and will also break automagic correlations (and data-validation of unique uuids)

This was caused by the switch of the mitre-intrusion-set to separate clusters for enterprise-attack, mobile-attack. My question is therefore: why exactly was everything moved to those 3 sub-clusters? Shouldn't some "common" things be kept together? (like: malware, tool, intrusion-set) While we could still split some others?

I know such a change would require implementation changes in MISP. But right now this seems wrong as we are breaking the UUID concept. Now you can't rely on a UUID to be unique.

Dec 02 '18 09:12 cvandeplas

The origin is pretty simple MITRE ATT&CK CTI. As MITRE decided to move to a STIX 2.0 model (which is a mistake IMHO), we basically converted the STIX files from the CTI repository into the separate clusters. Regarding the UUIDs, this is also another issue. We wanted to be sure that UUID will be kept consistent among the various export of the MITRE ATT&CK. They promise to do so. I fully agree that the common techniques should be kept together, I suppose the pivot ID should be more "TXXXX" external id than the UUID.

Looking at the recent discussion in OASIS CTI TC about UUIDs in STIX 2.0, I'm becoming more reluctant to do what we did in the past: Assuming that data providers will provide consistent UUID over the long-run/term... IMHO, it was a mistake I won't do again.

Maybe we should have a quick chat on how we should proceed with ATT&CK modelling especially ensuring safe transition between old/new release of the models, techniques and potentially new sub-techniques.

Dec 02 '18 10:12 adulau

Looking at it, it seems the easier would be to just consider the entities enterprise-attack, pre-attack and mobile-attack as meta values and merged into a single galaxy? This would be cleaner and save us if MITRE decides to change or extend the entities. What do you think?

Dec 02 '18 14:12 adulau

Following our call here was our current approach:

I'll merge again everything to generic clusters (like before, so one for intrusion-set) and will include the enterprise-attack , pre-attack, mobile-attack in one of the fields.
With this output we'll assess the impact on the other places it's used: MISP / Matrix,
we'll need to have an automatic conversion to fix existing data and replace those enterprise-attack , pre-attack, mobile-attack galaxies to the correct form

Dec 04 '18 10:12 cvandeplas

TODO:

[x] galaxy changes to bring it to the new version - by @cvandeplas https://github.com/MISP/misp-galaxy/pull/320
[ ] soft delete of tags with all the crap around it - by @iglocska
[x] migration tool to update all old style attack data - by @cvandeplas
[ ] include migration script in MISP update workflow - by @iglocska
[x] ATT&CK matrix updated - by @mokaddem
[ ] remove temporary aggregation - by @mokaddem

Jan 23 '19 12:01 cvandeplas

MITRE ATT&CK mapping issues with current MISP-galaxy implementation - uuid not unique