Plugin `_filter_compatible` check is slow on UNIX targets

[JSCU-CNI]

Currently commands such as target-query -l "*" example-target.img can take quite long on UNIX targets. Checks on Windows targets seem to perform significantly faster. Perhaps some performance profiling is in order.

[JSCU-CNI]

While #1247 did improve performance, the command is still relatively slow when compared to a Windows target.

$ time target-query -l=* -j windows-11-24h2.vmx | jq -r ".plugins.loaded | length"
174

real    0m21.101s
user    0m20.915s
sys     0m0.185s

$ time target-query -l=* -j ubuntu-server-24.04.2.vmx | jq -r ".plugins.loaded | length"
61

real    4m29.767s
user    4m29.010s
sys     0m0.747s

[JSCU-CNI]

The culprit seems to be somewhere in the NamespacePlugin load and/or (recursive) register logic. When we skip NamespacePlugin inside _filter_compatible using the patch below:

diff --git a/dissect/target/plugin.py b/dissect/target/plugin.py
index 878c08ca..9a6bbcf0 100644
--- a/dissect/target/plugin.py
+++ b/dissect/target/plugin.py
@@ -984,7 +984,7 @@ def _filter_tree_match(pattern: str, os_filter: str, show_hidden: bool = False)
 
 
 def _filter_compatible(
-    descriptors: list[FunctionDescriptor], target: Target, ignore_load_errors: bool = False
+    descriptors: list[FunctionDescriptor], target: Target, ignore_load_errors: bool = False, ignore_namespace_plugins: bool = True,
 ) -> Iterator[FunctionDescriptor]:
     """Filter a list of function descriptors based on compatibility with a target."""
     compatible = set()
@@ -998,6 +998,9 @@ def _filter_compatible(
         if descriptor.qualname in incompatible:
             continue
 
+        if ignore_namespace_plugins and descriptor.cls.__base__ == NamespacePlugin:
+            continue
+
         try:
             plugincls = load(descriptor)
         except Exception:

We get the following performance results:

$ time target-query -l=* -j windows-11-24h2.vmx | jq -r ".plugins.loaded | length"
157

real    0m18.176s
user    0m17.793s
sys     0m0.244s

$ time target-query -l=* -j ubuntu-server-24.04.2.vmx | jq -r ".plugins.loaded | length"
54

real    2m16.006s
user    2m15.590s
sys     0m0.399s

~For us, this seems to be an acceptable change, as we do not have a use case for a list of plugins with NamespacePlugins included - currently we post-process filter those out already to prevent duplicate record output in our processes. What do you think @Schamper?~

It seems some NamespacePlugins would be skipped entirely with this approach: NamespacePlugins without further inheritance, such as the mft plugin, which is of course not acceptable.

Perhaps we should register the plugins as we normally would in the Target instance, to prevent duplicate registering Namespace'd plugins (e.g. all browser plugins).

[Schamper]

It seems some NamespacePlugins would be skipped entirely with this approach: NamespacePlugins without further inheritance, such as the mft plugin, which is of course not acceptable.

This is a bug, see https://github.com/fox-it/dissect.target/issues/1180#issuecomment-2985205923.

[JSCU-CNI]

Performance seems to have significantly improved since testing in July. #1249 gives us another performance improvement, but the overall performance for unix and windows seems to be better now - even without that patch.

$ time target-query -l=* -j windows-11-24h2.vmx | jq -r ".plugins.loaded | length"
187

real    0m6.187s
user    0m6.078s
sys     0m0.116s

$ time target-query -l=* -j ubuntu-server-24.04.2.vmx | jq -r ".plugins.loaded | length"
94

real    0m7.848s
user    0m7.753s
sys     0m0.099s

With #1249 applied:

$ time target-query -l=* -j windows-11-24h2.vmx | jq -r ".plugins.loaded | length"
172

real    0m6.024s
user    0m5.913s
sys     0m0.118s

$ time target-query -l=* -j ubuntu-server-24.04.2.vmx | jq -r ".plugins.loaded | length"
84

real    0m4.967s
user    0m4.895s
sys     0m0.082s