MISP Connector 0 Events Processed

[OpenSecureCo]

Hey Team,

I am running OpenCTI version 5.3.7 via Docker and when attempting to import MISP events I am seeing data appear to be fetched but then see a "0 events have been processed" message within the Docker logs:

This also results in no data being brought into the platform:

Below are my connector settings:

connector-misp: image: opencti/connector-misp:5.3.7 deploy: placement: constraints: - node.role == worker privileged: true environment: - OPENCTI_URL=http://opencti:4000 - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN} - CONNECTOR_ID=cfa1b34c-4f07-4920-bd39-5b162eb8ae6e - CONNECTOR_TYPE=EXTERNAL_IMPORT - CONNECTOR_NAME=MISP - CONNECTOR_SCOPE=misp - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted) - CONNECTOR_UPDATE_EXISTING_DATA=false - CONNECTOR_LOG_LEVEL=info - MISP_URL=REDACTED - MISP_KEY=REDACTED - MISP_SSL_VERIFY=False # Required - MISP_DATETIME_ATTRIBUTE=timestamp # Required, filter to be used in query for new MISP events - MISP_CREATE_REPORTS=True # Required, create report for MISP event - MISP_CREATE_INDICATORS=True # Required, create indicators from attributes - MISP_CREATE_OBSERVABLES=True # Required, create observables from attributes - MISP_CREATE_OBJECT_OBSERVABLES=True # Required, create text observables for MISP objects - MISP_REPORT_CLASS=MISP Event # Optional, report_class if creating report for event - MISP_IMPORT_FROM_DATE=2020-01-01 # Optional, import all event from this date - MISP_IMPORT_TAGS= # Optional, list of tags used for import events - MISP_IMPORT_TAGS_NOT= # Optional, list of tags to not include - MISP_IMPORT_CREATOR_ORGS= # Optional, only import events created by those orgs (put the identifiers here) - MISP_IMPORT_CREATOR_ORGS_NOT= # Optional, do not import events created by those orgs (put the identifiers here) - MISP_IMPORT_OWNER_ORGS= # Optional, only import events owned by those orgs (put the identifiers here) - MISP_IMPORT_OWNER_ORGS_NOT= # Optional, do not import events owned by those orgs (put the identifiers here) - MISP_IMPORT_KEYWORD=opencti:import # Optional, search only events based on a keyword - MISP_IMPORT_DISTRIBUTION_LEVELS= # Optional, only import events with the given distribution levels - MISP_IMPORT_THREAT_LEVELS= # Optional only import events with the given threat levels - MISP_IMPORT_ONLY_PUBLISHED=False - MISP_IMPORT_WITH_ATTACHMENTS=False # Optional, try to import a PDF file from the attachment attribute - MISP_IMPORT_TO_IDS_NO_SCORE=10 # Optional, use as a score for the indicator/observable if the attribute to_ids is no - MISP_IMPORT_UNSUPPORTED_OBSERVABLES_AS_TEXT=False # Optional, import unsupported observable as x_opencti_text - MISP_INTERVAL=1 # Required, in minutes restart: always networks: - opencti_net depends_on: - opencti

I also have MISP events being tagged with the "opencti:import" tag:

I tried resetting the connector state a few times but still no luck. Initially it seems that the connector is able to fetch data but then nothing seems to be processed and uploaded to view within the WebUI. Anything I am missing here?

Appreciate the help and the awesome tool :)

Best Regards, Taylor

[0q1627]

I have experienced that something goes wrong if you define env variables without a value. Remove or comment-out those lines like this:

## - MISP_IMPORT_TAGS= # Optional, list of tags used for import events

[OpenSecureCo]

Hey @0q1627, thanks for getting back to me. Unfortunately I am still getting same results. I verified that MISP is returning JSON back as part of the request:

But still getting "0 events have been processed" message. Any other thoughts?

Thanks, Taylor

[bakhtawarw]

Did your issue get resolved? I am getting the same error.

[OpenSecureCo]

Hey @bakhtawarw - Unfortunately I am still getting the same error. I have tried resetting and deleting the connector and re running again but never seem to have luck. Hopefully the OpenCTI team will be able to assist further. Let me know if you ever find a solution.

Thanks

[SamuelHassine]

Hello everyone,

The latest version (5.3.7) should work well even with empty variables. But just to be sure, have you tried to remove all unnecessary parameters (empty)?

[SamuelHassine]

Configuration working in production for us:

misp:
  url: 'https://misp'
  reference_url: '' # Optional, will be used to create external reference to MISP event
  key: 'KEY' # Required
  ssl_verify: False # Required
  datetime_attribute: 'timestamp' # Required, filter to be used in query for new MISP events
  create_reports: True # Required, create report for MISP event
  create_indicators: True # Required, create indicators for attributes
  create_observables: True # Required, create observables for attributes
  create_object_observables: True # Required, create text observables for MISP objects
  report_class: 'misp-event' # Optional, report_class if creating report for event
  report_status: 'New' # New, In progress, Analyzed and Closed
  import_from_date: '2010-01-01 00:00:00' # Optional
  import_tags: '' # Optional, list of tags used for import events
  import_tags_not: '' # Optional, list of tags to not include
  import_creator_orgs: '' # Optional, only import events created by those orgs (put the identifiers here)
  import_creator_orgs_not: '' # Optional, do not import events created by those orgs (put the identifiers here)
  import_owner_orgs: '' # Optional, only import events owned by those orgs (put the identifiers here)
  import_owner_orgs_not: '' # Optional, do not import events owned by those orgs (put the identifiers here)
  import_owner_keyword: '' # Optional, search only events based on a keyword
  import_distribution_levels: '' # Optional, only import events with the given distribution levels
  import_threat_levels: '' # Optional only import events with the given threat levels
  import_only_published: False # Optional only import events published
  import_with_attachments: True # Optional, try to import a PDF file from the attachment attribute
  import_to_ids_no_score: 40 # Optional, use as a score for the indicator/observable if the attribute to_ids is no
  import_unsupported_observables_as_text: False # Optional, import unsupported observable as x_opencti_text
  interval: 1 # Required, in minutes

INFO:root:Initiate work for 799a5b1e-4965-4a25-a763-b61974a108e1
INFO:root:Connector has never run
INFO:root:Fetching MISP events with args: {"timestamp": "2010-01-01", "with_attachments": true, "limit": 50, "page": 1}
INFO:root:MISP returned 50 events.
INFO:root:Processing event 58dcfe62-ed84-4e5e-b293-4991950d210f
INFO:root:Sending event STIX2 bundle

[SamuelHassine]

By the way, we have modified the way empty variables are handled in connectors in the last version -5.3.7) so be sure to be in the latest version. If you still have problem, please copy/paste here your configuration.

[SamuelHassine]

@OpenSecureCo In your screenshot, it seems you've a tag "opencti:import", can you try to put in in the import_tags and not in the import keyword parameter?

[bakhtawarw]

Hi, this is the misp-connector configuration.

connector-misp: image: opencti/connector-misp:5.3.7 environment: - OPENCTI_URL=http://opencti:8080 - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN} - CONNECTOR_ID=bb9a05fb-719c-4463-ac31-448df5310f00 - CONNECTOR_TYPE=EXTERNAL_IMPORT - CONNECTOR_NAME=MISP - CONNECTOR_SCOPE=misp - CONNECTOR_CONFIDENCE_LEVEL=25 # From 0 (Unknown) to 100 (Fully trusted) - CONNECTOR_UPDATE_EXISTING_DATA=false - CONNECTOR_LOG_LEVEL=info - MISP_URL=http://10.0.2.4 # Required - MISP_REFERENCE_URL= # Optional, will be used to create external reference to MISP event (default is "url") - MISP_KEY=xxxxxxxxxxxxxx # Required - MISP_SSL_VERIFY=False # Required - MISP_DATETIME_ATTRIBUTE=timestamp # Required, filter to be used in query for new MISP events - MISP_CREATE_REPORTS=True # Required, create report for MISP event - MISP_CREATE_INDICATORS=True # Required, create indicators from attributes - MISP_CREATE_OBSERVABLES=True # Required, create observables from attributes - MISP_CREATE_OBJECT_OBSERVABLES=True # Required, create text observables for MISP objects - MISP_REPORT_CLASS=MISP Event # Optional, report_class if creating report for event - MISP_IMPORT_FROM_DATE=2020-01-01 # Optional, import all event from this date - MISP_IMPORT_TAGS= # Optional, list of tags used for import events - MISP_IMPORT_TAGS_NOT= # Optional, list of tags to not include - MISP_IMPORT_CREATOR_ORGS= # Optional, only import events created by those orgs (put the identifiers here) - MISP_IMPORT_CREATOR_ORGS_NOT= # Optional, do not import events created by those orgs (put the identifiers here) - MISP_IMPORT_OWNER_ORGS= # Optional, only import events owned by those orgs (put the identifiers here) - MISP_IMPORT_OWNER_ORGS_NOT= # Optional, do not import events owned by those orgs (put the identifiers here) - MISP_IMPORT_KEYWORD= # Optional, search only events based on a keyword - MISP_IMPORT_DISTRIBUTION_LEVELS=0,1,2,3 # Optional, only import events with the given distribution levels - MISP_IMPORT_THREAT_LEVELS=1,2,3,4 # Optional only import events with the given threat levels - MISP_IMPORT_ONLY_PUBLISHED=False - MISP_IMPORT_WITH_ATTACHMENTS=False # Optional, try to import a PDF file from the attachment attribute - MISP_IMPORT_TO_IDS_NO_SCORE=40 # Optional, use as a score for the indicator/observable if the attribute to_ids is no - MISP_IMPORT_UNSUPPORTED_OBSERVABLES_AS_TEXT=False # Optional, import unsupported observable as x_opencti_text - MISP_INTERVAL=1 # Required, in minutes restart: always depends_on: - opencti

These are the logs :

[OpenSecureCo]

@SamuelHassine Unfortunately same issue. See config below:

connector-misp: image: opencti/connector-misp:5.3.7 deploy: placement: constraints: - node.role == worker privileged: true environment: - OPENCTI_URL=http://opencti:4000 - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN} - CONNECTOR_ID=cfa1b34c-4f07-4920-bd39-5b162eb8ae6e - CONNECTOR_TYPE=EXTERNAL_IMPORT - CONNECTOR_NAME=MISP - CONNECTOR_SCOPE=misp - CONNECTOR_CONFIDENCE_LEVEL=25 # From 0 (Unknown) to 100 (Fully trusted) - CONNECTOR_UPDATE_EXISTING_DATA=true - CONNECTOR_LOG_LEVEL=info - MISP_URL=https://xxxxx - MISP_REFERENCE_URL=https://xxxxx - MISP_KEY=xxxxx - MISP_SSL_VERIFY=False # Required - MISP_DATETIME_ATTRIBUTE=timestamp # Required, filter to be used in query for new MISP events - MISP_CREATE_REPORTS=True # Required, create report for MISP event - MISP_CREATE_INDICATORS=True # Required, create indicators from attributes - MISP_CREATE_OBSERVABLES=True # Required, create observables from attributes - MISP_CREATE_OBJECT_OBSERVABLES=True # Required, create text observables for MISP objects - MISP_REPORT_CLASS=misp-event # Optional, report_class if creating report for event - MISP_IMPORT_FROM_DATE=2020-06-01 # Optional, import all event from this date - MISP_IMPORT_TAGS=opencti:import # Optional, list of tags used for import events - MISP_IMPORT_TAGS_NOT= # Optional, list of tags to not include - MISP_IMPORT_CREATOR_ORGS= # Optional, only import events created by those orgs (put the identifiers here) - MISP_IMPORT_CREATOR_ORGS_NOT= # Optional, do not import events created by those orgs (put the identifiers here) - MISP_IMPORT_OWNER_ORGS= # Optional, only import events owned by those orgs (put the identifiers here) - MISP_IMPORT_OWNER_ORGS_NOT= # Optional, do not import events owned by those orgs (put the identifiers here) - MISP_IMPORT_KEYWORD= # Optional, search only events based on a keyword - MISP_IMPORT_DISTRIBUTION_LEVELS= # Optional, only import events with the given distribution levels - MISP_IMPORT_THREAT_LEVELS= # Optional only import events with the given threat levels - MISP_IMPORT_ONLY_PUBLISHED=False - MISP_IMPORT_WITH_ATTACHMENTS=True # Optional, try to import a PDF file from the attachment attribute - MISP_IMPORT_TO_IDS_NO_SCORE=40 # Optional, use as a score for the indicator/observable if the attribute to_ids is no - MISP_IMPORT_UNSUPPORTED_OBSERVABLES_AS_TEXT=False # Optional, import unsupported observable as x_opencti_text - MISP_INTERVAL=1 # Required, in minutes restart: always networks: - opencti_net depends_on: - opencti

Still seems to be that MISP is returning data but OpenCTI is not processing it

Thanks for the help!

[OpenSecureCo]

@SamuelHassine I noticed some other things when putting the MISP connector into debug mode.

I see that the POST request includes a timestamp key with an epoch value:

If I make that same request to MISP via a curl, I see no data returned (curl -i -H "Accept: application/json" -H "content-type: application/json" -H "Authorization: xxxx" --data '{"tags": {"OR": ["opencti:import"]}, "timestamp": 1657312128, "with_attachments": true, "limit": 50, "page": 1}' -X POST https://xxxx --insecure) :

However, if I remove the timestamp as part of the request, I get data back (curl -i -H "Accept: application/json" -H "content-type: application/json" -H "Authorization: xxx" --data '{"tags": {"OR": ["opencti:import"]}, "with_attachments": true, "limit": 50, "page": 1}' -X POST https://xxx --insecure)

Does that epoch need to be converted to a normal timestamp? Does the timestamp field need to be removed?

Thanks, Taylor

[0q1627]

@bakhtawarw

• CONNECTOR_LOG_LEVEL=info

You can increase verbosity of the connector log to "debug" level. Anyway your issue is related to the response you get from MISP since the connector complains it is not receiving a JSON. Try the curl direct call as shown in a comment above:

curl -i -H "Accept: application/json" -H "content-type: application/json" -H "Authorization: xxx" --data '{"tags": {"OR": ["opencti:import"]}, "with_attachments": true, "limit": 50, "page": 1}' -X POST https://xxx --insecure

This is my configuration that is working

connector-misp2: image: opencti/connector-misp:5.3.7 environment: - OPENCTI_URL=http://opencti.local:8080 - OPENCTI_TOKEN=xxx - CONNECTOR_ID=yyy - CONNECTOR_TYPE=EXTERNAL_IMPORT - CONNECTOR_NAME=MISP_TLP - CONNECTOR_SCOPE=misp - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted) - CONNECTOR_UPDATE_EXISTING_DATA=False - CONNECTOR_LOG_LEVEL=debug - MISP_URL=http://10.X.Y.Z/ # Required - MISP_REFERENCE_URL= # Optional, will be used to create external reference to MISP event (default is "url") - MISP_KEY=zxyzxyzxy # Required - MISP_SSL_VERIFY=False # Required - MISP_DATETIME_ATTRIBUTE=timestamp # Required, filter to be used in query for new MISP events - MISP_CREATE_REPORTS=True # Required, create report for MISP event - MISP_CREATE_INDICATORS=True # Required, create indicators from attributes - MISP_CREATE_OBSERVABLES=True # Required, create observables from attributes - MISP_CREATE_OBJECT_OBSERVABLES=True # Required, create text observables for MISP objects - MISP_REPORT_CLASS="MISP Event" # Optional, report_class if creating report for event - MISP_IMPORT_FROM_DATE=2020-01-02 # Optional, import all event from this date - MISP_IMPORT_TAGS=tlp:green,tlp:white # Optional, list of tags used for import events - MISP_IMPORT_ONLY_PUBLISHED=False - MISP_IMPORT_WITH_ATTACHMENTS=False # Optional, try to import a PDF file from the attachment attribute - MISP_IMPORT_TO_IDS_NO_SCORE=40 # Optional, use as a score for the indicator/observable if the attribute to_ids is no - MISP_IMPORT_UNSUPPORTED_OBSERVABLES_AS_TEXT=False # Optional, import unsupported observable as x_opencti_text - MISP_INTERVAL=5 # Required, in minutes restart: always

Filtering works since I set MISP_IMPORT_TAGS instead of MISP_IMPORT_CREATOR_ORGS that never matched an Org. As you can see I removed all unused variables (I don't care anymore if that matters or not as long as I can import events that I want).

[OpenSecureCo]

@SamuelHassine One thing I have noticed is that manually running the curl command

curl -i -H "Accept: application/json" -H "content-type: application/json" -H "Authorization: xxxxx" --data '{"tags": {"OR": ["opencti:import"]}, "with_attachments": true, "limit": 50, "page": 1}' -X POST https://xxxx/events/restSearch --insecure

returns the json as expected but there is about a 18 second delay from the time of me making the curl request and the time that MISP responds with the json. Is it possible that the MISP connector is timing out before MISP has responded? So even though the search query is correct, the MISP container exists out before the MISP server responds?

Thanks, Taylor

[Tyrell20]

Same issue also for me with 5.3.7 version.

This is my configuration. I tried both with filters and without but the result is always the same, 0 events returned from my MISP instance.

      - CONNECTOR_NAME=MISP
      - CONNECTOR_SCOPE=misp
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_UPDATE_EXISTING_DATA=false
      - CONNECTOR_LOG_LEVEL=debug
      - MISP_URL=https://XXXXXX # Required
      - MISP_REFERENCE_URL= # Optional, will be used to create external reference to MISP event (default is "url")
      - MISP_KEY=XXXXXX # Required
      - MISP_SSL_VERIFY=False # Required
      - MISP_DATETIME_ATTRIBUTE=timestamp # Required, filter to be used in query for new MISP events
      - MISP_CREATE_REPORTS=True # Required, create report for MISP event
      - MISP_CREATE_INDICATORS=True # Required, create indicators from attributes
      - MISP_CREATE_OBSERVABLES=True # Required, create observables from attributes
      - MISP_CREATE_OBJECT_OBSERVABLES=True # Required, create text observables for MISP objects
      - MISP_REPORT_CLASS="MISP Event" # Optional, report_class if creating report for event
      - MISP_IMPORT_FROM_DATE=2022-07-20 # Optional, import all event from this date
      - MISP_IMPORT_ONLY_PUBLISHED=False
      - MISP_IMPORT_WITH_ATTACHMENTS=False # Optional, try to import a PDF file from the attachment attribute
      - MISP_IMPORT_TO_IDS_NO_SCORE=40 # Optional, use as a score for the indicator/observable if the attribute to_ids is no
      - MISP_IMPORT_UNSUPPORTED_OBSERVABLES_AS_TEXT=False #  Optional, import unsupported observable as x_opencti_text

MISP container Logs:

INFO:root:Fetching MISP events with args: {"timestamp": 1658924104, "limit": 50, "page": 1}
DEBUG:pymisp:POST - https://XXXXXXX/events/restSearch
DEBUG:pymisp:{"returnFormat": "json", "page": 1, "limit": 50, "withAttachments": 0, "metadata": 0, "timestamp": 1658924104, "enforceWarninglist": 0, "includeEventUuid": 0, "includeEventTags": 0, "sgReferenceOnly": 0, "includeContext": 0, "headerless": 0, "includeSightings": 0, "includeDecayScore": 0, "includeCorrelations": 0, "excludeDecayed": 0}
DEBUG:pymisp:{'User-Agent': 'PyMISP 2.4.159 - Python 3.10', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Cookie': 'MISP-5ed76dda-6dd0-43e5-a3cb-112c0a1ea115=9p56r33bt3q5rtbe575fs5dj24', 'Content-Length': '335', 'Authorization': 'XXXXXXX', 'content-type': 'application/json'}
DEBUG:urllib3.connectionpool:https://XXXXXXX:443 "POST /events/restSearch HTTP/1.1" 200 17
DEBUG:pymisp:{'response': []}
INFO:root:MISP returned 0 events.
INFO:root:Connector successfully run (0 events have been processed), storing last_run as 1658925255, and latest_event_timestamp as 1658924103

[ArifeVural]

I happen to have the same issue with OpenCTI not to be able to pull any events from MISP instance. Just wondering whether any progress on that. The curl cmd used by API can fetch entries when I ran it from shell. But, somehow the OpenCTI instance says 0 event returned.

When I first configured, a single event was fetched and then it stopped.

This is my config:

connector-misp: image: opencti/connector-misp:latest environment: - OPENCTI_URL=http://opencti:8080 - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN} - CONNECTOR_ID=${CONNECTOR_MISP_ID} - CONNECTOR_TYPE=EXTERNAL_IMPORT - CONNECTOR_NAME=MISP - CONNECTOR_SCOPE=misp - CONNECTOR_CONFIDENCE_LEVEL=3 - CONNECTOR_UPDATE_EXISTING_DATA=false - CONNECTOR_LOG_LEVEL=info - MISP_URL=${CONNECTOR_MISP_URL} # Required - MISP_KEY=${CONNECTOR_MISP_API} # Required - MISP_SSL_VERIFY=False # Required - MISP_CREATE_REPORTS=True # Required, create report for MISP event - MISP_REPORT_CLASS=MISP Event # Optional, report_class if creating report for event - MISP_IMPORT_FROM_DATE=2000-01-01 # Optional, import all event from this date - MISP_IMPORT_TAGS=opencti:import,type:osint # Optional, list of tags used for import events - MISP_INTERVAL=5 # Required, in minutes restart: always

Any help is appreciated.

Thanks.