Possible to add external connectors without docker?

[danielkelley743]

I've noticed that all of the pre-supplied templates come with docker-support.

Is there any documentation available on adding, and enabling external connectors without docker?

I'm running a standalone build without portainer.

[danielkelley743]

Found a solution.

[danielkelley743]

Ok, having a bit of a difficult time without docker:

INFO:root:Listing Threat-Actors with filters null.
ERROR:root:You must be logged in to do this.
OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration...
ERROR:root:You must be logged in to do this.

On every single connector launch.

[ckane]

You'll have to edit the config.yml files for the connectors you want to run and have it point to the OpenCTI URI, in addition to populating it with the credentials you wish to use to have the connector talk to OpenCTI. I have all of this stuff installed on a single AWS VM image, so I'll try logging into that later tonight and get you an example from there.

Once you have the config.yml populated, you should be able to execute the connector's main script file and it will start up in the foreground. I use systemd unit files that I've created to have systemd handle service management of each connector.

[danielkelley743]

@ckane

Thanks for this.

I've gone through several connector templates, configured the config.yml files however, seem to be getting an API error for every single one.

config.yml

opencti:
  url: 'http://localhost:8080'
  token: '830edb74-c952-4db7-a8b2-3f891a886f31'

connector:
  id: 'CISA-connector'
  type: 'EXTERNAL_IMPORT'
  name: 'CISA Known Exploited Vulnerabilities'
  scope: 'identity,vulnerability'
  confidence_level: 50 # From 0 (Unknown) to 100 (Fully trusted)
  update_existing_data: false
  run_and_terminate: false
  log_level: 'info'

cisa:
  catalog_url: 'https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json'
  interval: 2

command to launch:

root@NLDW2-D8:~/opencti/connectors/external-import/cisa-known-exploited-vulnerabilities/src# python3 cisa.py
INFO:root:Listing Threat-Actors with filters null.
ERROR:root:You must be logged in to do this.
OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration...

Just to clarify, I have installed requirements for python as well.

[ckane]

Are you sure OpenCTI is listening on port 8080 and not port 4000 on that system? ss -tunl to find out

[ckane]

My example is:

opencti:
  url: 'http://localhost:4000'
  token: '< redacted >'

connector:
  id: '1f0e9771-0fee-4bab-aa7f-fdd6c0a04d37'
  type: 'EXTERNAL_IMPORT'
  name: 'Common Vulnerabilities and Exposures'
  scope: 'identity,vulnerability'
  confidence_level: 75 # From 0 (Unknown) to 100 (Fully trusted)
  update_existing_data: false
  run_and_terminate: false
  log_level: 'info'

cve:
  nvd_data_feed: 'https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-recent.json.gz'
  history_data_feed: 'https://nvd.nist.gov/feeds/json/cve/1.1/'
  import_history: True # Import history at the first run (after only recent), reset the connector state if you want to re-import
  interval: 7 # In days, must be strictly greater than 1

[ckane]

Oh, id in your connector definition might need to be a UUID (from uuidgen) and not a plain string.

[danielkelley743]

OpenCTI is running & accessible on 8080.

root@NLDW2-D8:~/opencti/connectors/external-import/cisa-known-exploited-vulnerabilities/src# netstat -ltnp | grep -w ':8080'
tcp6       0      0 :::8080                 :::*                    LISTEN      48270/node
root@NLDW2-D8:~/opencti/connectors/external-import/cisa-known-exploited-vulnerabilities/src# netstat -ltnp | grep -w ':4000'
root@NLDW2-D8:~/opencti/connectors/external-import/cisa-known-exploited-vulnerabilities/src#

New config.yaml file:

opencti:
  url: 'http://localhost:8080'
  token: 'f2678239-870f-46d8-bf83-5569de236fb5"'

connector:
  id: '1f0e9771-0fee-4bab-aa7f-fdd6c0a04d37'
  type: 'EXTERNAL_IMPORT'
  name: 'Common Vulnerabilities and Exposures'
  scope: 'identity,vulnerability'
  confidence_level: 75 # From 0 (Unknown) to 100 (Fully trusted)
  update_existing_data: false
  run_and_terminate: false
  log_level: 'info'

cve:
  nvd_data_feed: 'https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-recent.json.gz'
  history_data_feed: 'https://nvd.nist.gov/feeds/json/cve/1.1/'
  import_history: True # Import history at the first run (after only recent), reset the connector state if you want to re-import
  interval: 7 # In days, must be strictly greater than 1

Not sure what's going on at the moment.

[danielkelley743]

root@NLDW2-D8:~/opencti/connectors/external-import/cisa-known-exploited-vulnerabilities/src# python3 cisa.py
INFO:root:Listing Threat-Actors with filters null.
ERROR:root:You must be logged in to do this.
OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration...
root@NLDW2-D8:~/opencti/connectors/external-import/cisa-known-exploited-vulnerabilities/src#

[ckane]

Oh, my config was for the cve module, not the cisa module (which I don't have configured)

[danielkelley743]

Oh, my config was for the cve module, not the cisa module (which I don't have configured)

root@NLDW2-D8:~/opencti/connectors/external-import/cve/src# python3 cve.py
INFO:root:Listing Threat-Actors with filters null.
ERROR:root:You must be logged in to do this.
OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration...

Same issue anyway.

[richard-julien]

Maybe an error in your token config

token: 'f2678239-870f-46d8-bf83-5569de236fb5"'

Looks like you put an extra double quote. Maybe should be

token: 'f2678239-870f-46d8-bf83-5569de236fb5'

[danielkelley743]

Maybe an error in your token config

token: 'f2678239-870f-46d8-bf83-5569de236fb5"'

Looks like you put an extra double quote. Maybe should be

token: 'f2678239-870f-46d8-bf83-5569de236fb5'

Didn't fix it.

It appears something is broken with openCTI itself. I've installed it 3 times, when you view connectors under the HTTP interface, there's this problem as well:

Error
An unknown error occurred. Please contact your administrator or the OpenCTI maintainers.

Wondering if this is a universal issue, there's nothing to work with.

[danielkelley743]

See below.

[danielkelley743]

This is what's being dropped in the buffer:

{"category":"APP","error":{"stacktrace":["Error: connect ECONNREFUSED 127.0.0.1:15672","at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1247:16)"]},"inner_relation_creation":0,"level":"error","message":"API Call","operation":"ConnectorsStatusQuery","operation_query":"query ConnectorsStatusQuery{...ConnectorsStatus_data}fragment ConnectorsStatus_data on Query{connectors{id name active auto connector_type connector_scope updated_at config{listen listen_exchange push push_exchange}}rabbitMQMetrics{queues{name messages messages_ready messages_unacknowledged consumers idle_since message_stats{ack ack_details{rate}}}}}","size":2,"time":11,"timestamp":"2022-08-24T09:42:52.422Z","type":"READ_ERROR","user":{"ip":"::ffff:82.7.49.131","referer":"http://REDACTED:8080/dashboard/data/connectors","user_id":"88ec0c6a-13ce-5e39-b486-354fe4a7084f"},"variables":{},"version":"5.3.7"}

EDIT: Fixed the HTTP issue, but still no luck with the connectors.

root@NLDW2-D8:~/opencti/connectors/external-import/cve/src# python3 cve.py
INFO:root:Listing Threat-Actors with filters null.
ERROR:root:You must be logged in to do this.
OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration...

[richard-julien]

For the graphql error you must check that you correctly deployed rabbitmq with the management plugin that allow OCTI to query information through rabbitmq API (in docker we use rabbitmq:3.10-management image).

For your connector, you have to check if OCTI port is accessible from the connector standpoint and verify that the token you use is a correct user token (corresponding to a token of a user profile)