Using debuggers as a dynamic analysis backend for capa

[yelhamer]

... I am really excited to hear that capa is adding the capacity to work with dynamic traces! I think dealing with API traces generated from a sandbox is a great start since it will provide more info than static analysis.

I think it would also be possible to run the feature extractor at instruction level on an execution trace recorded by tools like Windbg TTD/Reven/RR/Undo, etc. The immediate benefit is to see some behavior that we cannot see if the relevant code is encrypted. For example, if the sample uses a cyrpto function, but the code is encrypted, capa would not be able to see it directly. Besides, we can also see the concrete register/memory value at any time, this makes it possible to not only detect certain operations, but also obtain the data it actually operated on.

There would also be some challenges. For example, the trace can be quite long even for a few seconds of execution. Also, the boundary of function is not as clear when it comes to an execution trace. However, from my own experience of working with execution traces, despite the whole trace being very long, the unique instructions/basic blocks are pretty manageable. ...

Originally posted by @xusheng6 in #1517 (comment)