[CSV Mapper] It errors when processing csv file with new lines in one of the columns

[sweet-mentat]

Description

I tried to use CSV Mapper to ingest a csv with YARA rule indicators. I had a column for the pattern/YARA rule, and the YARA rule was formatted with white space and new lines. I created the configuration for the csv, and then tested it. It choked. The test resulted in an empty list, and here are some of the platform logs: {"category":"APP","cause":{"bytes":13,"code":"CSV_QUOTE_NOT_CLOSED","column":2,"columns":false,"comment_lines":0,"empty_lines":0,"header":false,"index":2,"invalid_field_length":0,"lines":1,"quoting":false,"records":0},"level":"error","line":"MALWARE NAME,yes,"rule Downloader_MALWARE_2","message":"Error parsing CSV line","timestamp":"2024-04-25T17:43:19.571Z","version":"6.0.9"} {"category":"APP","cause":{"bytes":0,"code":"INVALID_OPENING_QUOTE","column":0,"columns":false,"comment_lines":0,"empty_lines":0,"field":" author = ","header":false,"index":0,"invalid_field_length":0,"lines":1,"quoting":false,"records":0},"level":"error","line":" author = ""AUTHOR""","message":"Error parsing CSV line","timestamp":"2024-04-25T17:43:19.571Z","version":"6.0.9"}

I tried troubleshooting by writing a simple Python script to open the file using the csv library. It worked beautifully. It handled the column containing the new lines without any extra parameters: import csv with open("filename.csv") as csvfile: csvreader = csv.reader(csvfile) for row in csvreader: print(row)

At the Indicator list dashboard in OpenCTI, I exported some YARA indicators into a csv format. I tried to import that with the csv mapper. It had the same issue.

Here is the first line (with header) of my csv file: pattern,name,pattern_type,indicator_type "rule Downloader_MALWARE_2 { strings: $ = ""data/setup.exe"" $ = ""http[:]//somethings"" condition: uint16(0) == 0x5a4d and filesize < 10MB and all of them }",Downloader_MALWARE_2,yara,StixFile

Environment

1. OS (where OpenCTI server runs): Docker image
2. OpenCTI version: OpenCTI 6.0.9
3. OpenCTI client: frontend
4. Other environment details:

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Create Excel spreadsheet and add a column that contains new lines (such as a YARA pattern that contains new lines)
2. Create a CSV Mapper config to ingest csv file
3. Look at platform logs

Expected Output

CSV Mapper / CSV Importer should be able to ingest csv files with new lines.

[jborozco]

Hi @sweet-mentat I didn't manage to reproduce. Would it be possible for you to share with us your csv or a dummy version of the file so we can investigate further ?

[sweet-mentat]

test_indicator.csv Here is an example file

[sweet-mentat]

I also tried exporting yara indicators to a csv file from the indicators table, and then tried to import that with the csvmapper. It failed on that too.

[jborozco]

@sweet-mentat Sorry for the delay, I couldn't import your file but I was able to import a YARA indicator exported from the platform.

Can you share with us a screeen of your csv mapper configuration?

[sweet-mentat]

[sweet-mentat]

[sweet-mentat]

this is for the sample test_indicator.csv I provided.

[labo-flg]

I noticed this also, as https://github.com/OpenCTI-Platform/opencti/issues/7117

Having line feeds in a CSV is causing errors at extraction, even if the data string is between quotes