volatilityfoundation
Check layer dependency list based on a bool
In my code I had to explicitly call the del_layer function to remove a python reference count to an object saved in the layer.
In the runtime of the plugin many process layers were created and checking the depend_list each time I wanted to deleted my layer (not the process ones) took a LOT of time.
I used cProfile to determine what caused it to be that slow and 75% of my plugin's runtime was in the list comprehension line.
To address this issue I have added a flag which is defaulted to True (IE verify dependencies ) which in turn can disable this verification.
Currently no one uses the del_layer function.
I think this is a small symptom of a larger problem: the way layers work for processes in general.
For example when we want to access processes we call the add_process_layer function which creates a new layer EVERYTIME. Instead we should have a RAII like structure or a reference count and this way we won't need to create a new layer each time.
In volatility 2 the get_process_address_space function returned a new AS and the python garbage collector was responsible for cleaning it up when it was unused.
Currently we have unused stale layers that remain because they are referenced in the context.layers.
As I have said before, nobody calls del_layer so this layers remain as long as the context is alive.
I think we should change the way that works by not creating a new layer for a process AS if it already exists (least amount of changes this way).
Could you please generate a profile showing these calls (python -m cProfile -o del_layer.prof vol.py ...) so that I can look at how best to reduce the overhead?
My cProfile output containing the large overhead is in another network.
However the problem is when you keep all the layers "alive" and do a del_layer in the end.
For example, in the windows.cmdline plugin the get_cmdline function calls add_process_layer which creates a new layer. However that layer is not destroyed and as a result we leak a layer handle.
We should make layers scoped with a __exit__ and __enter__ and make them reference counted(maybe by process) so that no duplicate layers will be created for a process.
Example layer list at the end of ldrmodules plugin (which creates the process layer 4 times for each process)
['base_layer', 'memory_layer', 'layer_name', 'layer_name_Process4', 'layer_name_Process4_1', 'layer_name_Process4_2', 'layer_name_Process4_3', 'layer_name_Process92', 'layer_name_Process92_1', 'layer_name_Process92_2', 'layer_name_Process92_3', 'layer_name_Process324', 'layer_name_Process324_1', 'layer_name_Process324_2', 'layer_name_Process324_3', 'layer_name_Process416', 'layer_name_Process416_1', 'layer_name_Process416_2', 'layer_name_Process416_3', 'layer_name_Process492', 'layer_name_Process492_1', 'layer_name_Process492_2', 'layer_name_Process492_3', 'layer_name_Process504', 'layer_name_Process504_1', 'layer_name_Process504_2', 'layer_name_Process504_3', 'layer_name_Process584', 'layer_name_Process584_1', 'layer_name_Process584_2', 'layer_name_Process584_3', 'layer_name_Process600', 'layer_name_Process600_1', 'layer_name_Process600_2', 'layer_name_Process600_3', 'layer_name_Process632', 'layer_name_Process632_1', 'layer_name_Process632_2', 'layer_name_Process632_3', 'layer_name_Process744', 'layer_name_Process744_1', 'layer_name_Process744_2', 'layer_name_Process744_3', 'layer_name_Process760', 'layer_name_Process760_1', 'layer_name_Process760_2', 'layer_name_Process760_3', 'layer_name_Process752', 'layer_name_Process752_1', 'layer_name_Process752_2', 'layer_name_Process752_3', 'layer_name_Process864', 'layer_name_Process864_1', 'layer_name_Process864_2', 'layer_name_Process864_3', 'layer_name_Process948', 'layer_name_Process948_1', 'layer_name_Process948_2', 'layer_name_Process948_3', 'layer_name_Process964', 'layer_name_Process964_1', 'layer_name_Process964_2', 'layer_name_Process964_3', 'layer_name_Process440', 'layer_name_Process440_1', 'layer_name_Process440_2', 'layer_name_Process440_3', 'layer_name_Process912', 'layer_name_Process912_1', 'layer_name_Process912_2', 'layer_name_Process912_3', 'layer_name_Process1096', 'layer_name_Process1096_1', 'layer_name_Process1096_2', 'layer_name_Process1096_3', 'layer_name_Process1108', 'layer_name_Process1108_1', 'layer_name_Process1108_2', 'layer_name_Process1108_3', 'layer_name_Process1184', 'layer_name_Process1184_1', 'layer_name_Process1184_2', 'layer_name_Process1184_3', 'layer_name_Process1224', 'layer_name_Process1224_1', 'layer_name_Process1224_2', 'layer_name_Process1224_3', 'layer_name_Process1452', 'layer_name_Process1452_1', 'layer_name_Process1452_2', 'layer_name_Process1452_3', 'layer_name_Process1504', 'layer_name_Process1504_1', 'layer_name_Process1504_2', 'layer_name_Process1504_3', 'layer_name_Process1608', 'layer_name_Process1608_1', 'layer_name_Process1608_2', 'layer_name_Process1608_3', 'layer_name_Process1628', 'layer_name_Process1628_1', 'layer_name_Process1628_2', 'layer_name_Process1628_3', 'layer_name_Process1676', 'layer_name_Process1676_1', 'layer_name_Process1676_2', 'layer_name_Process1676_3', 'layer_name_Process1824', 'layer_name_Process1824_1', 'layer_name_Process1824_2', 'layer_name_Process1824_3', 'layer_name_Process1900', 'layer_name_Process1900_1', 'layer_name_Process1900_2', 'layer_name_Process1900_3', 'layer_name_Process1684', 'layer_name_Process1684_1', 'layer_name_Process1684_2', 'layer_name_Process1684_3', 'layer_name_Process2120', 'layer_name_Process2120_1', 'layer_name_Process2120_2', 'layer_name_Process2120_3', 'layer_name_Process2236', 'layer_name_Process2236_1', 'layer_name_Process2236_2', 'layer_name_Process2236_3', 'layer_name_Process2888', 'layer_name_Process2888_1', 'layer_name_Process2888_2', 'layer_name_Process2888_3', 'layer_name_Process2900', 'layer_name_Process2900_1', 'layer_name_Process2900_2', 'layer_name_Process2900_3', 'layer_name_Process3028', 'layer_name_Process3028_1', 'layer_name_Process3028_2', 'layer_name_Process3028_3', 'layer_name_Process2396', 'layer_name_Process2396_1', 'layer_name_Process2396_2', 'layer_name_Process2396_3', 'layer_name_Process3208', 'layer_name_Process3208_1', 'layer_name_Process3208_2', 'layer_name_Process3208_3', 'layer_name_Process3248', 'layer_name_Process3248_1', 'layer_name_Process3248_2', 'layer_name_Process3248_3', 'layer_name_Process3372', 'layer_name_Process3372_1', 'layer_name_Process3372_2', 'layer_name_Process3372_3', 'layer_name_Process3604', 'layer_name_Process3604_1', 'layer_name_Process3604_2', 'layer_name_Process3604_3', 'layer_name_Process3828', 'layer_name_Process3828_1', 'layer_name_Process3828_2', 'layer_name_Process3828_3', 'layer_name_Process2876', 'layer_name_Process2876_1', 'layer_name_Process2876_2', 'layer_name_Process2876_3', 'layer_name_Process3020', 'layer_name_Process3020_1', 'layer_name_Process3020_2', 'layer_name_Process3020_3', 'layer_name_Process2800', 'layer_name_Process2800_1', 'layer_name_Process2800_2', 'layer_name_Process2800_3', 'layer_name_Process2728', 'layer_name_Process2728_1', 'layer_name_Process2728_2', 'layer_name_Process2728_3', 'layer_name_Process1868', 'layer_name_Process1868_1', 'layer_name_Process1868_2', 'layer_name_Process1868_3', 'layer_name_Process4196', 'layer_name_Process4196_1', 'layer_name_Process4196_2', 'layer_name_Process4196_3', 'layer_name_Process4368', 'layer_name_Process4368_1', 'layer_name_Process4368_2', 'layer_name_Process4368_3', 'layer_name_Process4500', 'layer_name_Process4500_1', 'layer_name_Process4500_2', 'layer_name_Process4500_3', 'layer_name_Process4552', 'layer_name_Process4552_1', 'layer_name_Process4552_2', 'layer_name_Process4552_3', 'layer_name_Process4992', 'layer_name_Process4992_1', 'layer_name_Process4992_2', 'layer_name_Process4992_3', 'layer_name_Process5024', 'layer_name_Process5024_1', 'layer_name_Process5024_2', 'layer_name_Process5024_3', 'layer_name_Process5108', 'layer_name_Process5108_1', 'layer_name_Process5108_2', 'layer_name_Process5108_3', 'layer_name_Process3776', 'layer_name_Process3776_1', 'layer_name_Process3776_2', 'layer_name_Process3776_3', 'layer_name_Process3948', 'layer_name_Process3948_1', 'layer_name_Process3948_2', 'layer_name_Process3948_3', 'layer_name_Process2752', 'layer_name_Process2752_1', 'layer_name_Process2752_2', 'layer_name_Process2752_3', 'layer_name_Process4492', 'layer_name_Process4492_1', 'layer_name_Process4492_2', 'layer_name_Process4492_3', 'layer_name_Process5000', 'layer_name_Process5000_1', 'layer_name_Process5000_2', 'layer_name_Process5000_3', 'layer_name_Process2724', 'layer_name_Process2724_1', 'layer_name_Process2724_2', 'layer_name_Process2724_3', 'layer_name_Process4020', 'layer_name_Process4020_1', 'layer_name_Process4020_2', 'layer_name_Process4020_3', 'layer_name_Process2852', 'layer_name_Process2852_1', 'layer_name_Process2852_2', 'layer_name_Process2852_3', 'layer_name_Process3256', 'layer_name_Process3256_1', 'layer_name_Process3256_2', 'layer_name_Process3256_3', 'layer_name_Process3824', 'layer_name_Process3824_1', 'layer_name_Process3824_2', 'layer_name_Process3824_3', 'layer_name_Process1068', 'layer_name_Process1068_1', 'layer_name_Process1068_2', 'layer_name_Process1068_3', 'layer_name_Process4860', 'layer_name_Process4860_1', 'layer_name_Process4860_2', 'layer_name_Process4860_3', 'layer_name_Process2020', 'layer_name_Process2020_1', 'layer_name_Process2020_2', 'layer_name_Process2020_3', 'layer_name_Process4724', 'layer_name_Process4724_1', 'layer_name_Process4724_2', 'layer_name_Process4724_3', 'layer_name_Process4468', 'layer_name_Process4468_1', 'layer_name_Process4468_2', 'layer_name_Process4468_3', 'layer_name_Process3420', 'layer_name_Process3420_1', 'layer_name_Process3420_2', 'layer_name_Process3420_3', 'layer_name_Process4092', 'layer_name_Process4092_1', 'layer_name_Process4092_2', 'layer_name_Process4092_3', 'layer_name_Process4296', 'layer_name_Process4296_1', 'layer_name_Process4296_2', 'layer_name_Process4296_3', 'layer_name_Process4520', 'layer_name_Process4520_1', 'layer_name_Process4520_2', 'layer_name_Process4520_3', 'layer_name_Process3396', 'layer_name_Process3396_1', 'layer_name_Process3396_2', 'layer_name_Process3396_3', 'layer_name_Process3756', 'layer_name_Process3756_1', 'layer_name_Process3756_2', 'layer_name_Process3756_3', 'layer_name_Process5016', 'layer_name_Process5016_1', 'layer_name_Process5016_2', 'layer_name_Process5016_3', 'layer_name_Process4048', 'layer_name_Process4048_1', 'layer_name_Process4048_2', 'layer_name_Process4048_3', 'layer_name_Process3456', 'layer_name_Process3456_1', 'layer_name_Process3456_2', 'layer_name_Process3456_3', 'layer_name_Process2244', 'layer_name_Process2244_1', 'layer_name_Process2244_2', 'layer_name_Process2244_3', 'layer_name_Process4968', 'layer_name_Process4968_1', 'layer_name_Process4968_2', 'layer_name_Process4968_3', 'layer_name_Process360', 'layer_name_Process360_1', 'layer_name_Process360_2', 'layer_name_Process360_3', 'layer_name_Process3852', 'layer_name_Process3852_1', 'layer_name_Process3852_2', 'layer_name_Process3852_3', 'layer_name_Process5680', 'layer_name_Process5680_1', 'layer_name_Process5680_2', 'layer_name_Process5680_3', 'layer_name_Process4864', 'layer_name_Process4864_1', 'layer_name_Process4864_2', 'layer_name_Process4864_3', 'layer_name_Process3936', 'layer_name_Process3936_1', 'layer_name_Process3936_2', 'layer_name_Process3936_3', 'layer_name_Process1464', 'layer_name_Process1464_1', 'layer_name_Process1464_2', 'layer_name_Process1464_3', 'layer_name_Process2036', 'layer_name_Process2036_1', 'layer_name_Process2036_2', 'layer_name_Process2036_3', 'layer_name_Process5388', 'layer_name_Process5388_1', 'layer_name_Process5388_2', 'layer_name_Process5388_3', 'layer_name_Process4612', 'layer_name_Process4612_1', 'layer_name_Process4612_2', 'layer_name_Process4612_3', 'layer_name_Process1260', 'layer_name_Process1260_1', 'layer_name_Process1260_2', 'layer_name_Process1260_3', 'layer_name_Process1792', 'layer_name_Process1792_1', 'layer_name_Process1792_2', 'layer_name_Process1792_3', 'layer_name_Process1788', 'layer_name_Process1788_1', 'layer_name_Process1788_2', 'layer_name_Process1788_3', 'layer_name_Process5568', 'layer_name_Process5568_1', 'layer_name_Process5568_2', 'layer_name_Process5568_3', 'layer_name_Process2040', 'layer_name_Process2040_1', 'layer_name_Process2040_2', 'layer_name_Process2040_3', 'layer_name_Process5376', 'layer_name_Process5376_1', 'layer_name_Process5376_2', 'layer_name_Process5376_3']
Turning off the check is a no go. The check is there for a reason. If the state is too large, then better management of layers (adding and then removing them, rather than adding them all and then removing them all) would be a better solution. In the specific case, otherwise a pull request specifically for better management should be made. Making suggestions of __exit__ and __enter__ methods without an implementation puts the onus on the maintainers to develop and produce a solution to a problem that you say exists, but haven't provided the request cProfile data for (nor you example code that produces the issue). I'm quite happy to add appropriate deletion of layer calls to the various plugins, but it would be helpful to get those in another pull request please. As such I'm going to close this particular one since this isn't a solution, it's a flag to turn off a check that appears inconvenient.