'Ghost' watchlist - 2 watchlists in LA whereas there's only one in Sentinel

[Kaloszer]

Describe the bug 2 Watchlists exist with the same name/alias but different ID. No way to list it using az cli/rest api to be able to remove it.

Within environment 1 watchlist exists:

However when we look it up using _GetWatchlist('ETM') within LA we see that there are 2 😕

Note 2 different _DTItemIds !

When looking it up using az sentinel list/show - only the first one shows up - ergo no way to 'delete' it.

This causes any logic apps that rely on this being an 'object' and not an array of objects (e.g no foreach) -> this will only execute for one of the selected watchlists. This is an issue.

To Reproduce Steps to reproduce the behavior:

1. No idea how to reproduce this issue, seems unique

Expected behavior Watchlist does not have a 'twin' that cannot be removed.

Additional context Started occuring in a single environment/tenant, no other environments affected.

[v-muuppugund]

Hi @Kaloszer ,Could you please share more details on this issue with detailed issue replication steps with screen shots

[Kaloszer]

No idea how to replicate this, only have this in a single environment in production and can't remediate it.

[v-muuppugund]

No idea how to replicate this, only have this in a single environment in production and can't remediate it.

Hi @Kaloszer ,Could you please share team details who can reach them for replication steps in production environment, so can replicate at my end and fix the issue, Thanks.

[Kaloszer]

Did you even read the issue?

It just happened. No way to resolve this issue. No replication steps available. It should never happen ergo it's a bug.

[v-muuppugund]

Did you even read the issue?

It just happened. No way to resolve this issue. No replication steps available. It should never happen ergo it's a bug.

Hi @Kaloszer Got it,I have read the issue, want to replicate from my end,so can fix the issue,so asked it.

[Kaloszer]

Just because something does not have replication steps does not mean that it's not a bug. This incident just proves that the issue is inside of the workspace and had happened, it should never have happened. There's never a place in the sentinel workspace that 2 watchlists with the same name/alias exist as they will just be appended if one exists. This has happened ONCE in hundreds of runs of the 'remove > add watchlist' process which is needed to update bulk watchlist.

However there is no way to remediate this one-off issue because the tooling is incapable of doing so, the only way to fix it would be to purge that table, but I don't feel confident enough to do it because there's no what-if/dry-run capability in the purge API call.

Please route this somewhere to Microsoft as it seems you do not understand the issue at hand here.

[v-muuppugund]

Just because something does not have replication steps does not mean that it's not a bug. This incident just proves that the issue is inside of the workspace and had happened, it should never have happened. There's never a place in the sentinel workspace that 2 watchlists with the same name/alias exist as they will just be appended if one exists. This has happened ONCE in hundreds of runs of the 'remove > add watchlist' process which is needed to update bulk watchlist.

However there is no way to remediate this one-off issue because the tooling is incapable of doing so, the only way to fix it would be to purge that table, but I don't feel confident enough to do it because there's no what-if/dry-run capability in the purge API call.

Please route this somewhere to Microsoft as it seems you do not understand the issue at hand here.

Hi @Kaloszer ,got you, this is one of the scenario, we need to check the backend metrics and figure out the issue,will try to reach our team on this issue.

[Kaloszer]

Let me know if you need to have the sub/la/rg ids - I'll be able to provide them on Monday - [email protected]

[Kaloszer]

Just to confirm, when the original 'ETM' WL is removed, the latter is still there. CLI does not show that it's there - ergo can't delete it - can't fix the issue. Causing issues in production automations.

[hartshorne2]

+1 Also seeing this issue

[Kaloszer]

Any update, this is causing issues in production environment for a couple of weeks now...

[v-muuppugund]

Hi @Kaloszer ,Apologies for the delayed response,asked few more details over an email, Could you please share those details

[v-muuppugund]

+1 Also seeing this issue

Hi @hartshorne2 ,Could you please more details and its the same issue or different one and also share your email id to [email protected]

[Kaloszer]

@v-muuppugund Is there any update on this - the issue is still causing issues in our production environment and causing additional work for our agents.

[v-muuppugund]

@v-muuppugund Is there any update on this - the issue is still causing issues in our production environment and causing additional work for our agents.

Hi @Kaloszer , As got the details over an email, will be reaching concerned team on this issue,Will update you

[Kaloszer]

@v-muuppugund Any update?

[v-muuppugund]

@Kaloszer ,we are reaching out to the concerned team for this issue, once we receive an update on this, we will update you . Thanks!

[Kaloszer]

@v-muuppugund Any update?

[v-muuppugund]

@v-muuppugund Any update?

Hi @Kaloszer , As we are unable to repro the issue and also reached the concern team internally, If I have an update will post you over an email, so closing this issue

[Kaloszer]

How can this be closed if the issue is live in the current environment and I'm unable to fix it?

[v-muuppugund]

How can this be closed if the issue is live in the current environment and I'm unable to fix it?

Hi @Kaloszer , As we are unable to replicate the issue and don't have logs to verify,Please share couple of time slots will block calendar on this issue discussion and next steps

[Kaloszer]

@v-muuppugund Anything between 8-16 CEST works for me.

In addition we've also attempted to purge said ETM watchlist to no avail:

# Purge parameters
$table = "Watchlist"
$column = "WatchlistName"
$operator = "=="
# single value
$value = "ETM"
 
# Authenticate
Connect-AzAccount -SubscriptionId $SubscriptionId -TenantId $TenantId -ErrorAction Stop
 
$Params = @{
    ResourceGroupName = $ResourceGroupName
    WorkspaceName = $WorkspaceName
    Table = $table
    Column = $column
    OperatorProperty = $operator
    Value = $value
}
$Response = New-AzOperationalInsightsPurgeWorkspace @Params -Verbose -Debug
$operationId = $Response.OperationId
 
$Params = @{
    ResourceGroupName = $ResourceGroupName
    WorkspaceName = $WorkspaceName
    purgeId = $operationId
}

az rest --method GET --uri https://management.azure.com/subscriptions/1111/resourceGroups/rg/providers/Microsoft.OperationalInsights/workspaces/la/operations/purge-cd38e380-8151-463c-8749-e406b6555a62?api-version=2020-08-01
{
  "status": "completed"
}

ETM is still there:

EDIT: We've now also run it limited to the time, maybe that'll work.

[v-muuppugund]

@v-muuppugund Anything between 8-16 CEST works for me.

In addition we've also attempted to purge said ETM watchlist to no avail:

# Purge parameters
$table = "Watchlist"
$column = "WatchlistName"
$operator = "=="
# single value
$value = "ETM"
 
# Authenticate
Connect-AzAccount -SubscriptionId $SubscriptionId -TenantId $TenantId -ErrorAction Stop
 
$Params = @{
    ResourceGroupName = $ResourceGroupName
    WorkspaceName = $WorkspaceName
    Table = $table
    Column = $column
    OperatorProperty = $operator
    Value = $value
}
$Response = New-AzOperationalInsightsPurgeWorkspace @Params -Verbose -Debug
$operationId = $Response.OperationId
 
$Params = @{
    ResourceGroupName = $ResourceGroupName
    WorkspaceName = $WorkspaceName
    purgeId = $operationId
}

az rest --method GET --uri https://management.azure.com/subscriptions/1111/resourceGroups/rg/providers/Microsoft.OperationalInsights/workspaces/la/operations/purge-cd38e380-8151-463c-8749-e406b6555a62?api-version=2020-08-01
{
  "status": "completed"
}

ETM is still there:

EDIT: We've now also run it limited to the time, maybe that'll work.

Hi @Kaloszer .Apologies for delayed response,will connect on monday

[v-muuppugund]

Hi @Kaloszer ,As discussed on Monday i.e. 8Apr2024 ,reached our backend team over an email, will update over an email once have an update, we are closing your issue (https://github.com/Azure/Azure-Sentinel/issues/10107) as per our standard operating procedures. If you still need support for this issue, feel free to re-open at any time. Thank you for your co-operation!

[Kaloszer]

@v-muuppugund

Before closing this issue it was discussed on the call to provide a root cause, or an explaination on why this had happened. Otherwise this should not be closed

[v-muuppugund]

Hi @Kaloszer ,As discussed over call, reached our team internally over an email for having root cause from backend team once response received , will update you over an email,

[v-muuppugund]

Hi @Kaloszer ,As we are unable to raise ICM independently, so reached CSS team and they suggested support case from azure subscription, so they can assist you.

[v-sudkharat]

Hey @Kaloszer, We reached out to respective team for this issue, and as a response team mentioned that you need to raise a support case for this issue, as support team have the backend access to find the root cause of this issue. and if required the support team can forward the ticket to respective teams. So, we are closing this issue from GitHub. If you still need support for this issue, feel free to re-open it any time. Thank you for your co-operation.