\n in JSON string leads to unparsed event

[landauermax]

It is possible to parse the following event

{"@timestamp": "2024-04-18T11:00:42.606Z", "message": "abc", "winlog": {"user": {"name": "SYSTEM", "type": "Well Known Group", "identifier": "S-1-5-18", "domain": "NT AUTHORITY"}, "process": {"pid": 2316, "thread": {"id": 3928}}, "channel": "Microsoft-Windows-Sysmon/Operational", "computer_name": "atb-client02.aecid-testbed.com", "api": "wineventlog", "provider_name": "Microsoft-Windows-Sysmon", "opcode": "Info", "provider_guid": "{5770385f-c22a-43e0-bf4c-06f5698ffbd9}", "event_data": {"TargetUser": "AECID-TESTBED\\Bob", "SourceProcessId": "3396", "TargetImage": "C:\\Program Files\\Mozilla Firefox\\firefox.exe", "GrantedAccess": "0x1fffff", "SourceProcessGUID": "{816bf8e6-f731-6620-000f-000000000900}", "TargetProcessId": "1340", "SourceThreadId": "6960", "CallTrace": "C:\\windows\\SYSTEM32\\ntdll.dll+9e8f4|C:\\windows\\System32\\KERNELBASE.dll+58745|C:\\windows\\System32\\KERNELBASE.dll+54b93|C:\\windows\\System32\\ADVAPI32.dll+17d20|C:\\Program Files\\Mozilla Firefox\\firefox.exe+300bc|C:\\Program Files\\Mozilla Firefox\\firefox.exe+14228|C:\\Program Files\\Mozilla Firefox\\xul.dll+2619da3|C:\\Program Files\\Mozilla Firefox\\xul.dll+a62338|C:\\Program Files\\Mozilla Firefox\\xul.dll+a5f27f|C:\\Program Files\\Mozilla Firefox\\xul.dll+a5f53e|C:\\Program Files\\Mozilla Firefox\\xul.dll+faf3d|C:\\Program Files\\Mozilla Firefox\\xul.dll+188439e|C:\\Program Files\\Mozilla Firefox\\xul.dll+18834c3|C:\\Program Files\\Mozilla Firefox\\xul.dll+3b51f|C:\\Program Files\\Mozilla Firefox\\xul.dll+3b068|C:\\Program Files\\Mozilla Firefox\\nss3.dll+4f25|C:\\Program Files\\Mozilla Firefox\\nss3.dll+4dd1|C:\\windows\\System32\\ucrtbase.dll+21bb2|C:\\windows\\System32\\KERNEL32.DLL+17344|C:\\Program Files\\Mozilla Firefox\\mozglue.dll+da58|C:\\windows\\SYSTEM32\\ntdll.dll+526b1", "SourceUser": "AECID-TESTBED\\Bob", "RuleName": "-", "UtcTime": "2024-04-18 11:00:42.565", "SourceImage": "C:\\Program Files\\Mozilla Firefox\\firefox.exe", "TargetProcessGUID": "{816bf8e6-fd5a-6620-a50f-000000000900}"}, "record_id": 1403233, "task": "Process accessed (rule: ProcessAccess)", "version": 3, "event_id": "10"}}

but not this event where "message": "\n"

{"@timestamp": "2024-04-18T11:00:42.606Z", "message": "\n", "winlog": {"user": {"name": "SYSTEM", "type": "Well Known Group", "identifier": "S-1-5-18", "domain": "NT AUTHORITY"}, "process": {"pid": 2316, "thread": {"id": 3928}}, "channel": "Microsoft-Windows-Sysmon/Operational", "computer_name": "atb-client02.aecid-testbed.com", "api": "wineventlog", "provider_name": "Microsoft-Windows-Sysmon", "opcode": "Info", "provider_guid": "{5770385f-c22a-43e0-bf4c-06f5698ffbd9}", "event_data": {"TargetUser": "AECID-TESTBED\\Bob", "SourceProcessId": "3396", "TargetImage": "C:\\Program Files\\Mozilla Firefox\\firefox.exe", "GrantedAccess": "0x1fffff", "SourceProcessGUID": "{816bf8e6-f731-6620-000f-000000000900}", "TargetProcessId": "1340", "SourceThreadId": "6960", "CallTrace": "C:\\windows\\SYSTEM32\\ntdll.dll+9e8f4|C:\\windows\\System32\\KERNELBASE.dll+58745|C:\\windows\\System32\\KERNELBASE.dll+54b93|C:\\windows\\System32\\ADVAPI32.dll+17d20|C:\\Program Files\\Mozilla Firefox\\firefox.exe+300bc|C:\\Program Files\\Mozilla Firefox\\firefox.exe+14228|C:\\Program Files\\Mozilla Firefox\\xul.dll+2619da3|C:\\Program Files\\Mozilla Firefox\\xul.dll+a62338|C:\\Program Files\\Mozilla Firefox\\xul.dll+a5f27f|C:\\Program Files\\Mozilla Firefox\\xul.dll+a5f53e|C:\\Program Files\\Mozilla Firefox\\xul.dll+faf3d|C:\\Program Files\\Mozilla Firefox\\xul.dll+188439e|C:\\Program Files\\Mozilla Firefox\\xul.dll+18834c3|C:\\Program Files\\Mozilla Firefox\\xul.dll+3b51f|C:\\Program Files\\Mozilla Firefox\\xul.dll+3b068|C:\\Program Files\\Mozilla Firefox\\nss3.dll+4f25|C:\\Program Files\\Mozilla Firefox\\nss3.dll+4dd1|C:\\windows\\System32\\ucrtbase.dll+21bb2|C:\\windows\\System32\\KERNEL32.DLL+17344|C:\\Program Files\\Mozilla Firefox\\mozglue.dll+da58|C:\\windows\\SYSTEM32\\ntdll.dll+526b1", "SourceUser": "AECID-TESTBED\\Bob", "RuleName": "-", "UtcTime": "2024-04-18 11:00:42.565", "SourceImage": "C:\\Program Files\\Mozilla Firefox\\firefox.exe", "TargetProcessGUID": "{816bf8e6-fd5a-6620-a50f-000000000900}"}, "record_id": 1403233, "task": "Process accessed (rule: ProcessAccess)", "version": 3, "event_id": "10"}}

with the following config:

LearnMode: True

LogResourceList:
  - 'file:///home/ubuntu/windows/tmp.log'

Parser:
  - id: event_id
    type: VariableByteDataModelElement
    name: 'event_id'
    args: '0123456789'

  - id: computer_name
    type: AnyByteDataModelElement
    name: 'computer_name'

  - id: ts
    type: DateTimeModelElement
    name: 'timestamp'
    date_format: "%Y-%m-%dT%H:%M:%S.%fZ" # 2024-03-21T20:46:06.955Z

  - id: target_image
    type: AnyByteDataModelElement
    name: 'target_image'

  - id: source_image
    type: AnyByteDataModelElement
    name: 'source_image'

  - id: json
    start: True
    type: JsonModelElement
    name: 'model'
    allow_all_fields: True
    optional_key_prefix: "*"
    key_parser_dict:
      "@timestamp": ts
      winlog:
        computer_name: computer_name

Input:
        timestamp_paths: "/model/@timestamp/timestamp"

Analysis:
   - type: ParserCount
     id: ParserCount
     output_event_handlers:
       - "stpe"
     report_interval: 5

EventHandlers:
    - id: "stpe"
      type: "StreamPrinterEventHandler"
      json: True

    - id: 'stpefile'
      type: 'StreamPrinterEventHandler'
      json: true
      pretty: false
      output_file_path: '/tmp/anomalies.txt'

This is not expected since \n in strings of a JSON event should also be parsed with the AnyByteDataModelElement.

[ernstleierzopf]

This issue happens because special characters like \n or \t are not escaped, because they should rather be interpreted. However, these characters need to be escaped within string values of the json object.

This took quite a long time to debug..