New TI table fails to properly populate the 'ObservableKey' and 'ObservableValue' when stix pattern contains 'AND'

[nv-pipo]

Describe the bug Uploading a TI with 2 'values' separated by 'AND' doesn't populate the 'Observable*' fields. For example creating a TI with the stix string "[ipv4-addr:value = '193.233.254.9' AND domain-name:value = 'auth.linkedin.re.kr']", creates entries with no fields ObservableKey nor ObservableValue

To Reproduce Upload a new TI via the threat-intelligence-stix-objects:upload enpoint:

POST https://api.ti.sentinel.azure.com/workspaces/{workspace-id}/threat-intelligence-stix-objects:upload?api-version=2024-02-01-preview
Authorization: Bearer ...
Content-Type: application/json

{
    "sourcesystem": "test", 
    "stixobjects":[
        {
            "type": "indicator",
            "spec_version": "2.1",
            "id": "indicator--67e62408-e3de-4783-9480-f595d4fdae59",
            "name": "Test Indicator 'and' 3",
            "created": "2025-02-26T18:29:07.776Z", 
            "modified": "2025-02-26T18:29:07.776Z",
            "pattern": "[ipv4-addr:value = '1.1.1.1' AND domain-name:value = 'example.com']",
            "pattern_type": "stix",
            "valid_from": "2025-02-26T18:29:07.776Z",
            "valid_until": "2025-02-26T18:29:07.778Z",
        },
    ]
}

Query the result:

// new table
ThreatIntelIndicators
| where TimeGenerated <> now()
| where Id endswith "67e62408-e3de-4783-9480-f595d4fdae59"
| order by TimeGenerated desc
| project TimeGenerated, Created, Id, Pattern, ObservableKey, ObservableValue

The only field that contains the values is the 'Pattern' field, which contains the entire string. The 'ObservableKey' and 'ObservableValue' fields are empty.

Expected behavior

The 'ObservableKey' and 'ObservableValue' fields should be populated with the values from the TI. When 'OR' is used instead of 'AND' eg. the string "[ipv4-addr:value = '1.1.1.1' OR domain-name:value = 'example.com']" results in two entries in the table, one for each value, and the 'ObservableKey' and 'ObservableValue' fields are populated correctly for each entry.

In the previous table format, multiple fields with values are created (DomainName and NetworkIP).

// old table
ThreatIntelligenceIndicator
| where TimeGenerated <> now()
| where ExternalIndicatorId endswith "67e62408-e3de-4783-9480-f595d4fdae59"
| order by TimeGenerated desc

Don't get me wrong, we like that the new table uses only one field for the values, as it simplifies the use of the TI table...unfortunately, the current implementation doesn't work as expected when using 'AND' in the pattern.

[v-tsawant]

Hi @nv-pipo , Thanks for flagging this issue, we will investigate this issue and get back to you with some updates. Thanks!

[nv-pipo]

Hi @v-tsawant, what response are you waiting from our side? I reviewed the thread, but cannot see any request for further details.

Here are some stix patterns that we have recently received from a MISP subscription which fail to populate the "Observable*" fields:

[network-traffic:dst_ref.type = 'ipv4-addr' AND network-traffic:dst_ref.value = '179.158.105.44' AND network-traffic:dst_port = '443']

[file:name = 'help' AND (file:hashes.'MD5' = '1221b7e3a9b0950eb7ceb2a934e113b6' OR file:hashes.'SHA-1' = '85c57a8fe684f27cb7815d8514af83e4ba93775c' OR file:hashes.'SHA-256' = 'b4d8807f36a1efcc8fa86f81f4e0e2891280944cbb49e2d7b363c456cb8c62b4')]

[domain-name:value = 'goonboy.com' AND domain-name:resolves_to_refs[*].value = '111.67.3.187']

Clearly, we cannot control the patterns used by the upstream MISP service, so when the 'Observable*' is not populated we cannot use them.

[v-tsawant]

HI @nv-pipo , I sincerely apologize for the incorrect comment earlier. I would like to update you that we are actively investigating this issue and get back to you with some updates. Thanks!

[v-tsawant]

Hi @nv-pipo , Could you please suggest convenient time slots to this email id i.e. ([email protected]) for a team's meeting, so we can further troubleshoot the issue.

[nv-pipo]

Hi @v-tsawant, I've sent an email to the provided email address. The subject was: "Azure-Sentinel github issue 12075, request for meeting". Hope it doesn't go to your spam folder. Please let me know over here if it doesn't reach you. Best, Nicolas

[annasqc]

Hi @v-tsawant

We're running into this issue too and were wondering if there was a timescale for a fix being put in place?

Thanks,

Anna

[v-tsawant]

Hi @nv-pipo , The concerned team is already working on this issue, but we have not yet received a confirmed timeline for the fix. We will update you as soon as we have more information. Thank you for your patience.

[v-tsawant]

Hi @nv-pipo , Apologies for the delay in responding, as discussed with the team this is expected behavior. We will publish one row representing that indicator. The ObservableKey and ObservaleValue can only have one value in them. If the pattern has AND in it, we cannot preserve the semantic.

This is by design:

The ObservableKey and ObservableValue fields are only populated for simple equality comparisons, or an OR of equality comparisons in the pattern. If the pattern uses AND, other operators, or time-based STIX logic, these fields are left null intentionally, and only the Pattern field is populated.

If all your questions related to this GitHub issue have been addressed, would you like me to close the issue? Please let me know if you need any further assistance. Thanks!

[nv-pipo]

@v-tsawant ,

For us it is extremely important that Sentinel is able to digest STIX patterns with 'AND' clauses. In our case we have MISP providers that heavily use 'AND' clauses in their STIX patterns and we cannot change this. If Sentinel is not able to process these patterns, it will severely impact our ability to use the platform effectively.

For example, the following STIX pattern results in empty Observable* fields:

[network-traffic:dst_ref.type = 'ipv4-addr' AND network-traffic:dst_ref.value = '179.158.105.44' AND network-traffic:dst_port = '443']

[file:name = 'help' AND (file:hashes.'MD5' = '1221b7e3a9b0950eb7ceb2a934e113b6' OR file:hashes.'SHA-1' = '85c57a8fe684f27cb7815d8514af83e4ba93775c' OR file:hashes.'SHA-256' = 'b4d8807f36a1efcc8fa86f81f4e0e2891280944cbb49e2d7b363c456cb8c62b4')]

[domain-name:value = 'goonboy.com' AND domain-name:resolves_to_refs[*].value = '111.67.3.187']

Clearly, these patterns could be reduced to the most important parts to remove the use of 'AND' statements, but as previously mentioned, we cannot change the way our MISP providers structure their STIX patterns.

The most worrisome thing is that the previous table (ThreatIntelligenceIndicator) was able to digest/represent these STIX patterns with 'AND' clauses without any problems.

Will MS ever add support for 'AND' clauses in STIX patterns? As seen in other comments, we are not the only ones facing this issue.

[v-tsawant]

Hi @nv-pipo, Will update the same to our Team and based on their response will update to you. Thanks!

[cudeso]

Hello @v-tsawant . Is there a roadmap for Sentinel to support 'AND' clauses in STIX patterns?

[dso31]

Hi @v-tsawant @v-utpalkumar,

I would like to echo previous posters here about the outmost importance of this issue, this is a massive downgrade from previous functionality in the old api/table, it has effectivly broken our threat intel pipeline.

There is one example here that is extra egrigious, 'ip-address AND port' patters, this I cannot on my end split up into two patterns and still have a useful IOC. (unlike 'domain AND ip-address' combinations which I could split up), non working pattern example: [network-traffic:dst_ref.type = 'ipv4-addr' AND network-traffic:dst_ref.value = '1.2.3.4' AND network-traffic:dst_port = '8440']

Another example is 'ip-type AND ip-address' patterns, this should be simple to populate the the Observable* fields from, just ignore the ip-type. Non working pattern example: [network-traffic:dst_ref.type = 'ipv4-addr' AND network-traffic:dst_ref.value = '1.2.3.4']

These two examples are 100% of our IP-address related IOCs, we have 0 IP-addresses populated in the ObservableValue field. (out of 50491 unique patters with ip-addresses in them)

For us it would be an acceptable solution to add fields for each value type, similar to how the old table looked, that is something we at least can work with.

[inverzeio]

I've noticed Ingestion rule options have now an option in Conditions called "Connective", which allows both adding multiple Patterns or Values, which is pretty close to the functionality that'd solve the problem fairly easily, without a schema change.

However Actions are not enabling these, because not all Fields in Action are supported(e.g. Pattern, IndicatorValue), and you also cannot duplicate. Better Ingestion rule documentation with examples would be also very welcome.

I am not sure, if this contradicts any STIX primitives / requirements - if no, the suggestion remains, else please let me know and feel free to ignore. Thanks