[Bug] Can't delete Observables via API

[Neuro-HSOC]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Debian
OS version (client)	10
Virtualized Env.	False
Dedicated RAM	32 GB
vCPU	16
TheHive version / git hash	4.1.17-1
Package Type	DEB
Database	Cassandra
Index type	Elasticsearch
Attachments storage	HDFS
Browser type & version	Not applicable

Problem Description

It is not possible to delete observables via the API.

The observables can easily be found in the database (either from the Observable search in the UI or via thehive4py find_observables query) but it is not possible to delete them. Each request for deletion yields an 404 error in Requests and no observable is deleted.

In TheHive logs a org.thp.scalligraph.NotFoundError: Observable not found is raised at the same time. This IMHO doesn't make any sense though, since the observable was just found by a find_observable a second ago.

Steps to Reproduce

1. Create a case and add some observables
2. Search and delete them via API (e.g. with thehive4py and below code snippet)
3. Search again via Observable search and/or API and they are still there

#!/usr/bin/python3
import json
import sys
import requests
requests.packages.urllib3.disable_warnings()

from thehive4py.api import TheHiveApi
from thehive4py.query import *
from thehive4py.models import *
from thehive4py.exceptions import CaseTaskLogException

THEHIVE_URL = 'https://thehive'
THEHIVE_API_KEY = 'MYAPIKEY'

api = TheHiveApi(THEHIVE_URL, THEHIVE_API_KEY, cert=False)
list_to_clear = ['microsoft.com']

for item in list_to_clear:
    response = api.find_observables(query={"_and":[{"_field":"dataType","_value":"domain"},{"_field":"data","_value":item}]}, range='all')
    obs_data = response.json()
    for obs in obs_data:
        observable_id = obs['id']
        print("Deleting id", observable_id, "with value", obs['data'], api.delete_case_observable(observable_id))

Complementary information

Sample run of the above code:

foo@bar:~/Documents/development/hive_case_export$ python3 clear_observable.py 
Deleting id ~24985723040 with value microsoft.com <Response [404]>
Deleting id ~23880065240 with value microsoft.com <Response [404]>
...
Deleting id ~31825981640 with value microsoft.com <Response [404]>
Deleting id ~31580192800 with value microsoft.com <Response [404]>

Corresponding thehive logs for this:

2022-04-25 13:09:15,624 [ERROR] from org.thp.scalligraph.utils.Retry in application-akka.actor.default-dispatcher-31 [00008221|49ff3c22] uncaught error, not retrying
org.thp.scalligraph.NotFoundError: Observable not found
	at org.thp.scalligraph.traversal.TraversalOps$TraversalOpsDefs.getOrFail(TraversalOps.scala:146)
	at org.thp.thehive.controllers.v0.ObservableCtrl.$anonfun$delete$2(ObservableCtrl.scala:311)
	at org.thp.scalligraph.controllers.Entrypoint$EntryPointBuilder.$anonfun$authTransaction$2(Entrypoint.scala:77)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$tryTransaction$7(JanusDatabase.scala:241)
	at scala.util.Try$.apply(Try.scala:213)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$tryTransaction$6(JanusDatabase.scala:241)
	at scala.util.Try$.apply(Try.scala:213)
	at org.thp.scalligraph.utils.DelayRetry.withTry(Retry.scala:93)
	at org.thp.scalligraph.janus.JanusDatabase.tryTransaction(JanusDatabase.scala:238)
	at org.thp.scalligraph.controllers.Entrypoint$EntryPointBuilder.$anonfun$authTransaction$1(Entrypoint.scala:77)
	at org.thp.scalligraph.controllers.Entrypoint$EntryPointBuilder.$anonfun$auth$1(Entrypoint.scala:86)
	at org.thp.scalligraph.controllers.Entrypoint$EntryPointBuilder.$anonfun$asyncAuth$3(Entrypoint.scala:107)
	at org.scalactic.Good.fold(Or.scala:1229)
	at org.thp.scalligraph.controllers.Entrypoint$EntryPointBuilder.$anonfun$asyncAuth$2(Entrypoint.scala:107)
	at org.thp.scalligraph.DiagnosticContext$.$anonfun$withRequest$2(ContextPropagatingDisptacher.scala:102)
	at org.thp.scalligraph.DiagnosticContext$.saveDiagnosticContext(ContextPropagatingDisptacher.scala:108)
	at org.thp.scalligraph.DiagnosticContext$.withRequest(ContextPropagatingDisptacher.scala:99)
	at org.thp.scalligraph.controllers.Entrypoint$EntryPointBuilder.$anonfun$asyncAuth$1(Entrypoint.scala:107)
	at org.thp.scalligraph.auth.AuthSrvWithActionFunction$$anon$1.$anonfun$invokeBlock$2(AuthSrv.scala:91)
	at scala.Option.fold(Option.scala:251)
	at org.thp.scalligraph.auth.AuthSrvWithActionFunction$$anon$1.invokeBlock(AuthSrv.scala:90)
	at org.thp.scalligraph.auth.AuthSrvWithActionFunction$$anon$1.invokeBlock(AuthSrv.scala:87)
	at org.thp.scalligraph.auth.BasicAuthSrv$$anon$1.$anonfun$invokeBlock$1(BasicAuthSrv.scala:54)
	at scala.Option.fold(Option.scala:251)
	at org.thp.scalligraph.auth.BasicAuthSrv$$anon$1.invokeBlock(BasicAuthSrv.scala:54)
	at org.thp.scalligraph.auth.BasicAuthSrv$$anon$1.invokeBlock(BasicAuthSrv.scala:52)
	at org.thp.scalligraph.auth.SessionAuthSrv$$anon$1.$anonfun$invokeBlock$1(SessionAuthSrv.scala:98)
	at scala.Option.fold(Option.scala:251)
	at org.thp.scalligraph.auth.SessionAuthSrv$$anon$1.invokeBlock(SessionAuthSrv.scala:98)
	at org.thp.scalligraph.auth.SessionAuthSrv$$anon$1.invokeBlock(SessionAuthSrv.scala:95)
	at play.api.mvc.ActionBuilder$$anon$10.$anonfun$invokeBlock$2(Action.scala:408)
	at play.api.mvc.ActionBuilderImpl.invokeBlock(Action.scala:441)
	at play.api.mvc.ActionBuilderImpl.invokeBlock(Action.scala:439)
	at play.api.mvc.ActionBuilder$$anon$10.invokeBlock(Action.scala:408)
	at play.api.mvc.ActionBuilder$$anon$10.invokeBlock(Action.scala:404)
	at play.api.mvc.ActionBuilder$$anon$9.apply(Action.scala:379)
	at play.api.mvc.Action.$anonfun$apply$4(Action.scala:82)
	at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$4(Accumulator.scala:168)
	at scala.util.Try$.apply(Try.scala:213)
	at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$3(Accumulator.scala:168)
	at scala.Function1.$anonfun$andThen$1(Function1.scala:57)
	at scala.Function1.$anonfun$andThen$1(Function1.scala:57)
	at scala.Function1.$anonfun$andThen$1(Function1.scala:57)
	at play.api.libs.streams.StrictAccumulator.run(Accumulator.scala:199)
	at play.core.server.AkkaHttpServer.$anonfun$runAction$4(AkkaHttpServer.scala:417)
	at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
	at akka.http.scaladsl.util.FastFuture$.$anonfun$transformWith$3(FastFuture.scala:51)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at org.thp.scalligraph.ContextPropagatingDispatcher$$anon$1.$anonfun$execute$2(ContextPropagatingDisptacher.scala:57)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.thp.scalligraph.DiagnosticContext$.$anonfun$withDiagnosticContext$2(ContextPropagatingDisptacher.scala:93)
	at org.thp.scalligraph.DiagnosticContext$.saveDiagnosticContext(ContextPropagatingDisptacher.scala:108)
	at org.thp.scalligraph.DiagnosticContext$.withDiagnosticContext(ContextPropagatingDisptacher.scala:91)
	at org.thp.scalligraph.DiagnosticContext$$anon$2.withContext(ContextPropagatingDisptacher.scala:76)
	at org.thp.scalligraph.ContextPropagatingDispatcher$$anon$1.$anonfun$execute$1(ContextPropagatingDisptacher.scala:57)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
2022-04-25 13:09:15,624 [ERROR] from org.thp.scalligraph.models.Database in application-akka.actor.default-dispatcher-31 [00008221|49ff3c22] Exception raised, rollback (Observable not found)
2022-04-25 13:09:15,624 [WARN] from org.thp.scalligraph.ErrorHandler in application-akka.actor.default-dispatcher-31 [00008221|49ff3c22] DELETE /api/case/artifact/~31580192800 returned 404
2022-04-25 13:09:15,624 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-24 [00008221|] 1.2.3.4 DELETE /api/case/artifact/~31580192800 took 48ms and returned 404 57 bytes

[To-om]

I can't reproduce the problem. The error "Observable not found" occurs when the observable doesn't exist or when you don't have the permission to perform the requested action. Please ensure the user has the right "manageObservable" on the case the observable belong to. The search query also returns observables from alerts. For these observables, the user must have "manageAlert" right.

[Neuro-HSOC]

Just checked the permissions. They look fine:

Any advice how I can debug that further on our end? I can reliably reproduce it on our end with a lot of different Observable already in the system.

I will try it with a completely fresh case/observables too but even if that would work, it wouldn't solve all the old ones I already have in the system.

[To-om]

Do you have several organisations ? If the case comes from another organisation, ensure it has been shared with profile containing manageOrganisation. You can also try to remove observable using front-end.

[Neuro-HSOC]

Yeah, we have two organizations, our main and our testing one. But I get the issue with observables/cases being only present in our main orga. So, no sharing between organizations involved.

For example w3.org. In our test orga we don't have any observables:

In our main orga we have plenty and certainly way more than it would ever be feasible to be deleted from the UI:

Trying to delete them by API just gives 404 responses:

[To-om]

The observable are they in a case or in an alert ? Can run this command for some observable IDs and check output ?

curl -H 'Authorization: Bearer MYAPIKEY' 'https://thehive/api/v1/query' -H 'Content-type: application/json' -d '{"query": [{"_name": "getObservable", "idOrName": "~40964224"}, {"_name": "page", "from": 0, "to": 1, "extraData": ["isOwner", "shareCount", "permissions", "shares"]}]}' | jq .

[Neuro-HSOC]

Picked a few IDs from my screenshot and it looks like this:

foo@bar:~/Documents/development/hive_case_export$  curl -H 'Authorization: Bearer MYKEY' 'https://thehive/api/v1/query' -H 'Content-type: application/json' -d '{"query": [{"_name": "getObservable", "idOrName": "~34611490880"}, {"_name": "page", "from": 0, "to": 1, "extraData": ["isOwner", "shareCount", "permissions", "shares"]}]}' | jq .
[]

With paging and extraData I end up empty.

foo@bar:~/Documents/development/hive_case_export$  curl -H 'Authorization: Bearer MYKEY' 'https://thehive/api/v1/query' -H 'Content-type: application/json' -d '{"query": [{"_name": "getObservable", "idOrName": "~34611490880"}]}' | jq .
[
  {
    "_id": "~34611490880",
    "_type": "Observable",
    "_createdBy": "node-red@local",
    "_createdAt": 1643117281071,
    "dataType": "domain",
    "data": "w3.org",
    "startDate": 1643117281071,
    "tlp": 2,
    "tags": [],
    "ioc": false,
    "sighted": false,
    "reports": {},
    "extraData": {}
  }
]

Just requesting by ID gives me an observable as I would have expected it.

Also tried with just the paging part {"query": [{"_name": "getObservable", "idOrName": "~34611490880"}, {"_name": "page", "from": 0, "to": 1}]} and then I get the same as querying just for the ID without any additions.

[To-om]

Do you have any error in logs ? Can you try to POST /api/v1/admin/check/Observable/global/trigger with admin user ?

[Neuro-HSOC]

Doing the request including the extraData parameter, indeed gives an error:

2022-05-06 16:18:00,846 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-23 [0003cd7d|] 127.0.0.1 POST /api/v1/query took 67ms and returned 200 
2022-05-06 16:18:00,864 [WARN] from org.thp.scalligraph.utils.Retry in application-akka.actor.default-dispatcher-23 [|] An error occurs (java.lang.IllegalArgumentException: The provided traverser does not map to a value: v[34611490880]->[JanusGraphVertexStep(IN,[ShareObservable],vertex), TraversalFilterStep([JanusGraphVertexStep(IN,[OrganisationShare],vertex), HasStep([~id.eq(4296)])]), JanusGraphVertexStep(OUT,[ShareProfile],vertex)]), retrying (1)

Without extraData there are no errors. The check observable trigger giving me this result {"type":"NotFoundError","message":"/api/v1/admin/check/Observable/global/trigger"} and a 404 in the logs 2022-05-06 16:53:36,264 [WARN] from org.thp.scalligraph.ErrorHandler in application-akka.actor.default-dispatcher-34 [|] POST /api/v1/admin/check/Observable/global/trigger returned 404

Assuming I read the code of the Router.scala file for the v1 router correctly, /api/v1/admin/check/Observable/trigger should do fine for the same function too, shouldn't it. Trying that command I get back a 204 No Content and this in the logs 2022-05-06 16:56:36,656 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-25 [0003d136|] 127.0.0.1 GET /api/v1/admin/check/Observable/trigger took 72ms and returned 204 0 bytes

Doing a /api/v1/admin/check/stats a few minutes later, I get that back in regards to observables. Not sure whether it is of any help to trace the error:

    "Observable": {
        "needCheck": false,
        "duplicateTimer": false,
        "duplicateStats": {
            "global": {
                "duplicate": 0,
                "duration": 3,
                "iteration": 1
            },
            "last": {
                "duplicate": 0,
                "duration": 3
            },
            "lastDate": 1650635459222
        },
        "globalStats": {
            "global": {
                "duration": 21680375,
                "Observable-Organisation-extraLinks": 611,
                "iteration": 3,
                "Observable-organisationIds-removeOrphan": 162223,
                "Observable-relatedId-removeOrphan": 611
            },
            "last": {
                "Observable-organisationIds-removeOrphan": 41,
                "duration": 5542369
            },
            "lastDate": 1651698642276
        },
        "globalCheckRequestTime": 0
    },

[To-om]

There was many orphan. How to do create the observable to have so many observable that are not linked to organisation/case ? Do they come from migration ? Do you often delete case/alert ? Now, your python script should work.

[To-om]

If the observable comes from an analyzer report, I can't be removed. This could be the main reason on why you get a 404.

[Neuro-HSOC]

The DB certainly has quite some updates/migrations behind it. We started our instance with the first version of Hive4 and been updating since, including running into DB migration issues more than once. E.g. what we described in #1878 . In the end we managed to perform the upgrade to 4.1.17 based on the immenseTerm Processing and forceDropRebuildIndex Cassandra settings described in the 4.1.16 release blog article. As part of that journey we three times deleted old cases on a larger scale to reduce DB size.

The observable creation is mainly done via analyzers plus some automation using NodeRed. The flow we utilize here, is NodeRed triggering a bunch of analyzers automatically, taking the artifacts of the analyzer job, filtering out quite a bunch of them and then actively add the remaining artifacts as observables to the corresponding case. While the analyzers are configured to not create observables on their own, this certainly gives us a lot of artifacts from cortex jobs, which don't make it into a "full" observable connected to a case. So, likely that is the source of our undeletable observables. :+1:

But wouldn't that in the long run create a huge overhead in the database? :thinking: Is there any recommended way to do some DB housekeeping to get rid of those? Regularly run these checkers perhaps? Maybe something on cortex side as well, to clean up analyzer jobs?

Lastly, is there any way to filter the analyzer related observable out in the find_observables query? If so, I would give that a try to confirm observables not related to an analyzer report can be deleted.

[Neuro-HSOC]

I extended my cleaning script a little to do some counting based on the response codes.

Looks like there are a couple of obversables getting deleted and a lot which don't. Which would fit, that something like w3.org in our setup shouldn't make it into an actual observable but very likely is getting picked up in a lot of EMLParser/FileInfo results of mails and HTML files.

Then I get the 404 error in general, but the question remains how to ensure I don't drag the unused artifacts around in my database for the next eternity. :laughing: