[Email] Import with CSV attachment ingestion

[NetEyes]

Hi,

Feature Request: Email import and ingest CSV feeds attachment.

Summary: We are receiving daily threat intel via email with (intel feeds as CSV format) and it would be great to have this ingested automatically to OpenCTI for further analysis and correlation.

Thanks.

[nor3th]

Since someone else on Slack also mentioned interest in an e-mail ingestion connector, I could look into writing one. Conceptually it could look like this:

1. a new EXTERNAL_IMPORT connector for fetching emails via POP3/IMAP/EWS and uploading the entire e-mail or just the attachments to a newly created report
2. The import-report INTERNAL_IMPORT connector can easily be extended to parse uploaded email files (like .eml or .msg).

If the import-report connector is configured to automatically parse newly uploaded files (via CONNCETOR_AUTO), the email ingestion process can be accomplished automatically.

I am not a huge fan of storing the credentials for the email access in the docker-compose configuration. Storing them as docker secret might be a bit safer way

[timebotdon]

I have been working on an IMAP connector that does exactly this (ingests csv attachments based on a ioc/description format). You've got a good point @nor3th , I would'nt store creds on the config too.

[nor3th]

Do you want to maybe write the email connector? I would start with a bit broader use case though than just attached csv files.

Use case 1: The relevant information is covered in the e-mail body itself Use case 2: The relevant information is attached to the e-mail as a csv, pdf, ... file

Either way the approach would be like written above, where the EXTERNAL_IMPORT email connector collects the email and then either uploads the content of the email body to a new OpenCTI report or the attachments are uploaded. Another INTERNAL_IMPORT connector then parsers the uploaded files of the report.

It would be great if the connector can figure out where the relevant information is stored dynamically. If not, then it has to be told via the config in advance.

I think the Intelmq team did a pretty similar approach. Check out their code: https://github.com/certtools/intelmq/tree/develop/intelmq/bots/collectors/mail

[timebotdon]

I do have a working prototype and would love to contribute to the repo!

The prototype can currently fulfuill both use cases as stated, albeit in a pretty limited and restrictive capacity.

1. The email message body is parsed using regex for the relevant IOC information. The email subject would be designated a report title, appending to a new report bundle

2. The IOCs attached in csv/txt/pdf/xlsx formats are extracted and appended to the report

The report is then pushed to OpenCTI.

However I'm not exactly too sure if there are standards/processing/formating I should follow with regards to OpenCTI dev CI/CD!

[nor3th]

Hey

That would be amazing if you could contribute that connector. If you haven't already looked at the connector building guide, here's a link: https://www.notion.so/HowTo-Build-your-first-connector-06b2690697404b5ebc6e3556a1385940

Ideally keep the connector simple (KISS) and the code should be easily readable. Unfortunately the CI only checks currently performed are flake8 --ignore=E,W && black . concerning the code layout. I am thinking about implementing tests for connectors, but I haven't managed time wise yet. So please test your code manually for now.

If you want push your current prototype to your fork and I can take a look at it.

We'd prefer to have a micro service layout, hence I would avoid working on implementations which are already implemented in the import-report connector. Ideally the IMAP connector should be able to fetch emails, create a new report and store the text of the email or the attachment as a file in the report. As a next step the import-report connector can get to work and extract all IOCs and further information. If you are missing anything in the import-report connector either submit a PR or create an issue and I'll try to add the capability soon.

[SamuelHassine]

Hello @timebotdon,

Please open a PR for your working code, we can also complete it if necessary. It's actually a very good idea!

Thanks for your work!

Kind regards, Samuel