Optimise bitmap crashdump parsing, add support for types 0x02 and 0x06

[paulkermann]

Parsing bitmap crashdumps was significantly slowed down by _context.object() array creation. I optimised get_buffer_long() and get_buffer_char() in framework/symbols/windows/extensions/crash.py, by returning Python types instead of volatility objects.

Additionally, crashdump bitmap of type 5 is the same as type 6, so I added it to the list of supported types.

[ikelos]

Thanks for this! Unfortunately, I'm quite uncomfortable returning none-volatility objects in this one specific case when all of our other code returns volatility objects. It somewhat breaks expectations, and to take it to extremes, eveything could be faster if we parsed it all with struct.unpack. I'm not against optimizing these things, but I'd do it as different methods, rather than changing the API if there isn't a way to speed up the context.object call. Is there a specific call in object that takes a particularly long time, is it just how often this called (would caching the result help?). It looks like the get_buffer call could definitely be memoized if that would improve matters, and I think I'd be happier with that than the python object return.

Also @iMHLv2 this is crashdump functionality so I dunno if you or @awalters had a view on the changes?

[paulkermann]

is there a specific call in object that takes a particularly long time

The problem's not with one specific object, it's with the amount of objects.

I have seen 800MB crashdumps with 0x100000 long bitmaps, creating a volatility object for each just takes a lot of time. I think in this case and generally in dump-format parsing, using non-volatility objects for some things is valid. The array being read is just a bitmap and we don't need the whole vol.offset and vol.type_name infrastructure for each of those entries.

[paulkermann]

I'm working on dump_type 0x02 support, I think this will greatly improve performance there too

[paulkermann]

Any progress on this?

[awalters]

Hi @paulkermann, are you able to share any memory samples we can use for testing the commits?

[paulkermann]

@awalters where should I upload the memory dump?

[paulkermann]

@awalters attached is a sample Windows 11 type 0x06 crashdump

[awalters]

@paulkermann Thank you for submitting the memory sample! How frequently are you working with kernel memory samples? Did you have any issues with any of the plugins that leverage user memory? Have you been able to verify the patch on both 32 and 64 bit samples?

[paulkermann]

@awalters This type of dump only contains kernel memory and the user memory of the process that caused the dump. No other user process memory is available. User mode plugins will only work for the process that crashed the PC.

I am having trouble setting up computers for testing. If you have a 32 bit windows 10 PC then you could use Process Hacker's Create live dump... functionality in Tools to create a dump and test it. It think it should work though and If it does not I would be glad to add support to it as well.

[paulkermann]

@ikelos @awalters I added support for type 0x02 now. Also I have tested that it works on Windows 7 dumps (type 0x02) and windows 10 dumps (type 0x06) for both x86 and x64.

[paulkermann]

I created and used these crashdump samples for debugging purposes

[digitalisx]

All right @paulkermann, now you'll be able to test your code to see if it works properly. :)

[paulkermann]

Hey, any update on this? @ikelos

[ikelos]

I'm afraid not, this is still outstanding on @awalters and @iMHLv2 to look into...

[paulkermann]

@ikelos @awalters I've seen this mentioned in an issue - any updates on merging the pull request? :')

[ikelos]

No progress on this front yet I'm afraid. I'm still not keen on supporting partial dumps (kernel/process only dumps), but if there's anything that's needed for improving full dump support, if you can split that out into its own PR we can probably get that merged much more quickly...

@awalters any other comments from your side?