[ThreatMatch] Connector needs improvements

[helene-nguyen]

Description

Corrections Required

• A simple connector run generated more than 100 errors in the connector errors view
  • “Only stix-core-relationship can be created through this method”. The connector is ingested STIX2.1 formatted data that contains invalid or unrecognized relation-type like: “associated_content” (malware → campaign)
• The connector is ingested Threat Actor and not intrusion-set entity to model adversaries

• Most knowledge is modeled as labels -> too many labels

As the connector directly ingests data exposed in STIX 2.1 format, modifications can be complicated to implement. 2 possibilities:

• Ask the supplier to change the way it models things
• Reconvert entities identified as problematic at connector level

Other Improvements:

• “CONNECTOR_UPDATE_EXISTING_DATA” deprecated option that need to be removed
• Not compatible with Duration period, usage of a specific THREATMATCH_INTERVAL variable
• Entity descriptions are in HTML format and not in markdown
• Documentation doesn’t:
  • describe what type of entities are ingested

Environment

1. OS (where OpenCTI server runs): { e.g. Mac OS 10, Windows 10, Ubuntu 16.4, etc. }
2. OpenCTI version: { e.g. OpenCTI 1.0.2 }
3. OpenCTI client: { e.g. frontend or python }
4. Other environment details:

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. { e.g. Run ... }
2. { e.g. Click ... }
3. { e.g. Error ... }

Expected Output

Actual Output

Additional information

Screenshots (optional)

[pietrocapece]

Heya, just looking for some clarification on this comment here: "Not compatible with Duration period, usage of a specific THREATMATCH_INTERVAL variable" Are you asking me to change the THREATMATCH_INTERVAL variable? does the OPENCTI connector template provide some default variable for this?

[pietrocapece]

@flavienSindou @helene-nguyen I am attempting to implement these changes you have recommended but require a clarification that I asked above.

[Lhorus6]

Hello @pietrocapece, You can have a look in the connector template files to see how it should be implemented. You can see:

• how the configuration is retrieved : HERE
• how the connector leverage it : HERE

Nothing more AFAIK. @helene-nguyen, @flavienSindou can you confirm ?

[pietrocapece]

I have submitted a draft PR for your review: https://github.com/OpenCTI-Platform/connectors/pull/4074 Please let me know if these were the changes you were looking for

[romain-filigran]

To be implemented to move towards a first "Verification"

• [ ] Replace all "associated_content" relationships by "related-to" relationships. "associated_content" relationship is not defined in STIX2.1 and consequently, it's not supported by OpenCTI. Implement the logic provided in the PR.

• [ ] Add a connector option to model 'threat-actor' as 'intrusion-set'. Most of our users and connectors model threat actors in the form of Intrusion-Set, we need to add this option (true by default) to deduplicate information.

• [ ] Remove any labels: STIX objects contain too many labels representing various entities, such as attack patterns, regions, countries, etc. Importing such large labels can break the platform and is especially wasteful when processing data. Remove all labels before implementing better label processing logic.

[pdamoune]

Hello @helene-nguyen @romain-filigran @pietrocapece

After a check up of the connector, I noticed all of this steps in order to pass the connector as verified (non exhaustive list)

I added what you just described to it @romain-filigran and the bug with description field https://github.com/OpenCTI-Platform/connectors/pull/4074

Checklist Item	Category	Status
The connector manages to start up without error	Before	✅
No use of interval → duration_period instead	Deprecate	❌
No use of the customised property x_opencti_report_status	Deprecate	✅
No use of the helper logger helper.log_{level}()	Deprecate	❌
"No use of variable CONFIDENCE_LEVEL "	Deprecate	❌
No use of variable helper.api…	Deprecate	❌
No use of variable UPDATE_EXISTING_DATA	Deprecate	❌
Check configuration variables	Configuration	❌
Config file samples	Configuration	❌
docker-compose.yml file	Configuration	❌
Dockerfile	Configuration	✅
README - Add Verified table	Configuration	❌
README - Make sure it’s complete	Configuration	❌
requirements.txt	Configuration	✅
~~Use of cleanup_inconsistent_bundle~~	Code	❌
Author is well referenced on entities	Code	❌
Define CONNECTOR_TYPE directly in the application	Code	❌
Knowledge is modeled and have relationships	Code
PydanticSettings - Make sure of default_factory=True in Field	Code	✅
PydanticSettings - Make sure of extra=”allow” in model_config	Code	✅
Use a clear Error Handling	Code	❌
Use a Traceback in the main	Code	❌
Use absolute import	Code	✅
Use external references to the source portal	Code	✅
Use Marking (TLP) on entities	Code	❌
Use of pycti.[STIX_ENTITY].generate_id()	Code	❌
Use proper date formatting AND time zone	Code	❌
Ensure auto backpressure is implemented	Code	❌
Ensure the connector doesn’t generate errors in OpenCTI	Code	❌
Work 1/4 - Ensure work is initiated	Code	✅
Work 2/4 - Ensure work is closed properly	Code	✅
Work 3/4 - Ensure initiate_work_id is only trigger if bundle is not empty	Code	❌
Work 4/4 - Ensure work total is not incrementing through time	Code	❌
Description field missing https://github.com/OpenCTI-Platform/connectors/issues/4110	Bug	❌
"associated_content" -> "related-to"	Bug	❌
'threat-actor' as 'intrusion-set'	Bug	❌
Remove any labels	Bug	❌

@pietrocapece, as I am full time on this from now on, I am going to integrate the fixes and what I meant in my comments on your PR, to the PR to come very soon. I will then close your PR once everything is validated if it's ok for you