Feature Request: Custom Queries

[WildDogOne]

Hey everyone,

I've been playing around with Sigma for our QRadar instllation. And I've noticed that some queries just don't work (or at least I think that's how it is) For example: if Value is Greater than X or: if property is in reference set and so on. Now I am aware that QRadar is a bit of a special child so I don't think it would make sense to implement everything that QRadar does.

But might it be of interest to have the possibility to input statements directly in the language needed?

Example:

detection:
  selection:
    username: X
  timeframe: 1d
  condition: selection and confidence > 80

Right now I just put inline comments because we use sigma primarily for documentation so I do this:

detection:
  selection:
    username: X
    #confidence > 80
  timeframe: 1d
  condition: selection

I hope someone understands what I mean 😄

Regards,

Linus

[moutonjr]

+1 Was definitely looking for this feature. I'm using Sigma for documentation also, and on Sentinel resulting query must be lintered, like with projection & agregations, what Sigma doesn't do.

Yet it's heavily SIEM-dependent.

If I can broaden definition : having a field to store SIEM-dependent query so that sigmac doesn't compile and just takes the field. (override).

It strongly weakens portability of rule but enable leverage other Sigma capacities (out of compilation I mean).

For instance, having:

detection:
  sentinel:
    kql: "SecurityEvents | where EventID == 4720 | summarize count(*) by ComputerName" # dummy

[jshlbrd]

this is also keeping me from adopting sigma; we ended up creating our own version of detection as code for the platform we use. sigma would have been a good fit for the use case if custom queries were supported.

[moutonjr]

Hey @DemonWareXT @jshlbrd Figured out that RX validator and spec are quite vague at this subject.

We added custom field on it (and custom SIEM params) to make it work. no problem.

Of course it's not published because against the very purpose of Sigma : being SIEM-independant. Yet useful for analytics using Sigma.

Shall this be closed?

[WildDogOne]

@moutonjr I am way too stupid to understand this 😄 I've checked the sigma schema, and of course it's easily possible to add new custom values to the YAML, but they wouldn't be translated into the respective search query or did I understand something wrong?

[Neo23x0]

@moutonjr

Shall this be closed?

I'm not yet if I'd like to close this issue. I think many others could have the same issues and then find out that they could add their own platform-specific queries in additional optional fields.

I think, I'll close this issue after adding a special section to the README for clarification.

The

...
[arbitrary custom fields]

suffix in the format specification isn't visible enough.

https://github.com/SigmaHQ/sigma/wiki/Specification#structure

[sifex]

We shoehorned this into our own sigmac wrapper a year or so ago. It didn't get a huge amount of uptake, but here's what we learnt at least when implementing across SIEMs.

• We used the special keyword query to indicate that the wrapper should not use the sigmac converter and instead only use the raw query instead.
• The wrapper also did a lookahead for where the logsource shouldn't be prepended. In this case, | tstats shouldn't have index=win* source="WinEventLog:System prefix attached, but a regular Splunk search should have it. Each backend would essentially need to be aware of this.

title: 'An example rule'
level: low
logsource:
  product: windows
  service: system
detection:
  query:
    splunk: >
      | tstats count where index="*" by _time, index
    sumologic: >
      (other query)
  condition: query
  timeframe: 30m
fields:
- index

In hindsight, I think a detection.engine to be (sigma|raw) (where sigma would use the selection and condition fields and raw would use the name of the SIEM) would be better suited. Maybe even have it apart of the new schema field in v2.

imho: I think Sigma should include this in v2 as a feature, but forbid it to be included in the common rule repos, and instead have it as a breakglass options for teams to be able to use locally. I can imagine once you make raw query sharing the norm, you end up having two problems

• it completely does away with the Sigma project's ethos in favor for more complete detection content repos (such as splunk/security_content)
• it fractures the community back into their SIEM-specific query way of thinking, making sharing less automatic for Detection Engineers and Analysts

[thomaspatzke]

This would totally break with the goal of Sigma that rules are vendor and environment agnostic. I also don't see the advantage of using Sigma as container for raw queries. This are the use cases we intentionally left for individual extensions of Sigma.

Something I thought about is to extend the pySigma processing pipelines to the query generation stage and allow addition of read query parts while conversion.

Regarding the specific use cases of numeric comparisons, pySigma has modifiers like gt, lt etc for this purpose.

[sifex]

I think the thing I forgot to mention, both why we made the wrapper and why this feature is resonating with some of the community is that Sigma is acting as a proxy for being able to centrally manage and version control alerts across multiple clients & SIEMs, or at least that's how we used it.

Analysts could start utilising gitops to PR changes for rules & defeats across clients.

edit: word

[WildDogOne]

hahaha @sifex I did pretty much the exactly same dirty hack for kusto a few weeks back. and yeah I agree with @thomaspatzke this is of course way against the point of Sigma, however sometimes it's better to have a dirty hack documentation in Sigma, comapared to nothing at all.

I will at some point try and get my hands on pysigma, but no time right now to extend for our needs

[thomaspatzke]

however sometimes it's better to have a dirty hack documentation in Sigma, comapared to nothing at all.

Disagree 😉 The problem is: documenting some dirty hacks in the Sigma specification means that we take over some kind of ownership for it. People will use it, identify issues, request enhancements or just ask how to use it in their specific case. All this creates additional load on the core maintainers to handle these requests or frustration on the side of users if requests are not handled. Therefore, each enhancement to the core must be considered very well. This is also the reason why backends, conversion configs (aka processing pipelines) etc. are now separated into dedicated projects, that not even must be located below the SigmaHQ umbrella. There are lots of great ideas that end up as one-shot contributions that have then to be maintained by the long-term maintainers. This is not sustainable.

And now comes the constructive part 😉

Sigma by itself is extensible by design and pySigma can be extended easily without touching the library itself. In the specific case you can add arbitrary attributes for your own purposes. Everything that pySigma doesn't handles lands in the custom_attributes dict of a SigmaRule object. There are lots of locations in backends where you can hook into the conversion process from rule to query, pick up these custom attributes and do whatever you want. E.g. for the use case @sifex mentioned above, the method convert_rule could be overriden by some code that checks for an attribute and uses this as query instead performing the normal conversion. This could be also implemented as separate project with mixin classes that gets configured by class attributes, like many things are already configured by such attributes in the TextQueryBackend.