Processing pipeline guidance

[kartone]

Looking for guidance: I wonder how I can create a pipeline yml file that simply drops unsupported fields present the rule that there aren't supported by my backend. For example, Hashes fields is not supported by my telemetry. This is the rule I'm trying to convert and I want to drop the entire Hashes section and Imphash fields. Thank you!

This is the pipeline YML file I'm using as command line argument in the sigma convert command.

 priority: 100
 transformations:
     # Drops the Hashes field which is specific to Sysmon logs
     - id: hashes_drop_sysmon-specific-field
       type: drop_detection_item
       field_name_conditions:
         - type: include_fields
           fields:
           - Hashes
       rule_conditions:
       - type: logsource
         product: windows
         category: process_creation

However it seems not working:

sigma convert -t sentinel_one_pq -p s1_pipeline.yml  ../sigma/rules/windows/process_creation/proc_creation_win_pua_process_hacker.yml
Parsing Sigma rules  [####################################]  100%
Error while conversion: Invalid SigmaDetectionItem field name encountered: Hashes. This pipeline only supports the following fields:
{CommandLine}, {Company}, {CurrentDirectory}, {Description}, {DestinationHostname}, {DestinationIp}, {DestinationPort}, {Details}, {Image}, {ImageLoaded}, {Imphash}, {IntegrityLevel}, {OriginalFileName}, {ParentCommandLine}, {ParentImage}, {ParentProcessId}, {PipeName}, {ProcessId}, {Product}, {Protocol}, {Publisher}, {QueryName}, {SourceFilename}, {SourceIp}, {SourcePort}, {TargetFilename}, {TargetObject}, {TerminalSessionId}, {User}, {answer}, {dst_ip}, {dst_port}, {md5}, {query}, {record_type}, {sha1}, {sha256}, {src_ip}, {src_port}

[joshnck]

This is a problem with the backend and not pySigma - please see this link to the section of code that is supporting that error generation: https://github.com/7RedViolin/pySigma-backend-sentinelone-pq/blob/24123bcba1cc4ee425a55d3ef6b46a64c90109b7/sigma/pipelines/sentinelone_pq/sentinelone_pq.py#L26

[kartone]

@joshnck I see. I understand that translation_dict is meant to translate field naming, I can't understand how I can discard certain fields. Thanks for your help.

[joshnck]

@joshnck I see. I understand that translation_dict is meant to translate field naming, I can't understand how I can discard certain fields. Thanks for your help.

So your pipeline logic is correct but there is an error in the handling of the fields in the backend that is giving you the issue. I'm not sure how you'll approach that with the backend but if you consider your logic mapped to a different backend like Splunk it works as expected:

Check here for an example: sigconverter.io

[kartone]

Ok. Indeed I've remapped the OriginalFileName to another supported field within my local backend source code, solving this issue.

Still, the backend is unable to follow the pipeline. Should I fire an issue on its repo?

[joshnck]

That would be my recommendation. I'm not very familiar with that backend but the way it is handling these errors in my cli is really unusual. AFAIK you cannot overwrite failure items and so you need to modify it at the first time it occurs.

[thomaspatzke]

I have two answers for you 😉

The first one is: don't do this! It's strongly discouraged to just drop parts of the rule that are not supported by the targeted backend or data model. The reason is that dropping parts of the detection changes its intended logic, which results in:

• more false positives if the dropped conditions are AND-linked, the good case. Or
• false negatives also known as the false sense of security if the dropped conditions are OR-linked, which is the worse case.

In the case of the rule you referred the worse case will happen. The detection can be better bypassed by attackers.

The possibility for processing pipelines to raise errors in such cases was introduced in pySigma to warn the users of it or tools based on it about this fact. In Sigma CLI you can use --skip-unsupported to skip over such rules.

The second answer is: if you know what you're doing and want to do it anyways, you should use the drop_detection_item transformation together with an include_field_condition similar to this part of the CrowdStrike processing pipeline.

[kartone]

Hey @thomaspatzke, thanks for your reply. I'm aware of the downsides of doing this and, as of now, this project is into its first experimental phase. I'm going to use SentinelOne that does not support imphash anyway.

The second answer is: if you know what you're doing and want to do it anyways, you should use the drop_detection_item transformation together with an include_field_condition similar to this part of the CrowdStrike processing pipeline.

Based on you advice, I modified the pipeline like this but still it stucks when it encounter Imphash field. Can you please advice? Many thanks.

[kartone]

Eventually I was able to drop unsupported fields, look at my pipeline now.

Basically I had to add the unsupported fileds inside the translation_dict and create single DropDetectionItemTransformation() sections for each field. It looks a bit odd to me, but I'm not able to troubleshoot more due to my limited knowledge on how to read structured python projects.

Ideally, I'd want to improve the SentinelOne backend. Looking for advices. :)