SwapExceptions may actually be compressedmemory pages

[MrAdz350]

Getting the below error when running the windows.malfind.Malfind plugin on an image with the following info:

Kernel Base	0xf8054ccbd000
DTB	0x1ad000
Symbols	file:///Users/user/Downloads/volatility3/volatility/symbols/windows/ntkrnlmp.pdb/E0093F3AEF15D58168B753C9488A4043-1.json.xz
primary	0 WindowsIntel32e
memory_layer	1 FileLayer
KdVersionBlock	0xf8054d0e73c8
Major/Minor	15.18362
MachineType	34404
KeNumberProcessors	2
SystemTime	2019-11-13 16:52:07
NtSystemRoot	C:\Windows
NtProductType	NtProductWinNt
NtMajorVersion	10
NtMinorVersion	0
PE MajorOperatingSystemVersion	10
PE MinorOperatingSystemVersion	0
PE Machine	34404
PE TimeDateStamp	Mon Apr 14 21:36:50 2104

user-mbp:volatility3 user$ python3 vol.py -v -f ~/Desktop/Memory\ Images/VAE_CRT_RX.vmem windows.malfind.Malfind
Volatility 3 Framework 1.0.0-beta.1
INFO     root        : Volatility plugins path: ['/Users/user/Downloads/volatility3/volatility/plugins', '/Users/user/Downloads/volatility3/volatility/framework/plugins']
INFO     root        : Volatility symbols path: ['/Users/user/Downloads/volatility3/volatility/symbols', '/Users/user/Downloads/volatility3/volatility/framework/symbols']
INFO     volatility.plugins.yarascan: Python Yara module not found, plugin (and dependent plugins) not available
INFO     volatility.plugins.yarascan: Python Yara module not found, plugin (and dependent plugins) not available
INFO     volatility.plugins.yarascan: Python Yara module not found, plugin (and dependent plugins) not available
INFO     volatility.plugins.yarascan: Python Yara module not found, plugin (and dependent plugins) not available
INFO     volatility.plugins.windows.verinfo: Python pefile module not found, plugin (and dependent plugins) not available
INFO     root        : The following plugins could not be loaded (use -vv to see why): volatility.plugins.windows.callbacks, volatility.plugins.windows.svcscan, volatility.plugins.windows.vadyarascan, volatility.plugins.windows.verinfo, volatility.plugins.yarascan
INFO     volatility.framework.automagic: Detected a windows category plugin
INFO     volatility.framework.automagic: Running automagic: ConstructionMagic
INFO     volatility.framework.automagic: Running automagic: LayerStacker
INFO     volatility.framework.automagic: Running automagic: WinSwapLayers
INFO     volatility.framework.automagic: Running automagic: WintelHelper
INFO     volatility.framework.automagic: Running automagic: KernelPDBScanner
INFO     volatility.schemas: Dependency for validation unavailable: jsonschema

PID	Process	Start VPN	End VPN	Tag	Protection	CommitCharge	PrivateMemory	Hexdump	Disasm
Traceback (most recent call last):
  File "vol.py", line 10, in <module>
    volatility.cli.main()
  File "/Users/user/Downloads/volatility3/volatility/cli/__init__.py", line 442, in main
    CommandLine().run()
  File "/Users/user/Downloads/volatility3/volatility/cli/__init__.py", line 269, in run
    renderers[args.renderer]().render(constructed.run())
  File "/Users/user/Downloads/volatility3/volatility/cli/text_renderer.py", line 160, in render
    grid.populate(visitor, outfd)
  File "/Users/user/Downloads/volatility3/volatility/framework/renderers/__init__.py", line 196, in populate
    for (level, item) in self._generator:
  File "/Users/user/Downloads/volatility3/volatility/framework/plugins/windows/malfind.py", line 115, in _generator
    for vad, data in self.list_injections(self.context, self.config["nt_symbols"], proc):
  File "/Users/user/Downloads/volatility3/volatility/framework/plugins/windows/malfind.py", line 92, in list_injections
    vadinfo.VadInfo.protect_values(context, proc_layer_name, symbol_table), vadinfo.winnt_protections)
  File "/Users/user/Downloads/volatility3/volatility/framework/symbols/windows/extensions/__init__.py", line 410, in get_protection
    protect = self.u.VadFlags.Protection
  File "/Users/user/Downloads/volatility3/volatility/framework/objects/__init__.py", line 689, in __getattr__
    raise AttributeError("{} has no attribute: {}.{}".format(agg_name, self.vol.type_name, attr))
AttributeError: StructType has no attribute: nt_symbols1!<anonymous-tag>.VadFlags

[ikelos]

Hiya, there was a small issue (#135) that might have affected symbols generated in the last two days, could you please remove the file /Users/user/Downloads/volatility3/volatility/symbols/windows/ntkrnlmp.pdb/E0093F3AEF15D58168B753C9488A4043-1.json.xz and then re-run the plugin and check if that changes the output please?

[MrAdz350]

Done, no dice and same error as above. I only downloaded them initially about an hour ago too.

[ikelos]

Mmmmm, we did do some work that might affect <anonymous-tag>, but I'm not convinced it will help. You can try the improve-pdbconv branch to see if that resolves the issue, but otherwise we'll probably need a memory image to be able to recreate the problem. 5:S There's also someone experiencing issues with the same memory kernel/symbol table in issue #135, if they can help?

[mattwhatkins]

@MrAdz350 and I are working together on a shared project. I also get the same error :(

[ikelos]

Ah, I don't suppose you'd be able to recreate the error with a memory image you'd be willing to share with us, would you? Remote debugging is usually trickier and more time consuming... 5;)

[ikelos]

Hmmm, so swapping to the improve-pdbconv branch generates a better pdb file (which won't fail with the nt_symbols1!<anonymous-tag>.VadFlags error), so I'd recommend rebuilding the symbol table file regardless (swap to the branch, wipe out the old file, and rerun any plugin against the image).

The error now generated is:

Traceback (most recent call last):
  File "vol.py", line 10, in <module>
    volatility.cli.main()
  File "/home/mike/workspace/volatility3/volatility/cli/__init__.py", line 442, in main
    CommandLine().run()
  File "/home/mike/workspace/volatility3/volatility/cli/__init__.py", line 269, in run
    renderers[args.renderer]().render(constructed.run())
  File "/home/mike/workspace/volatility3/volatility/cli/text_renderer.py", line 159, in render
    grid.populate(visitor, outfd)
  File "/home/mike/workspace/volatility3/volatility/framework/renderers/__init__.py", line 196, in populate
    for (level, item) in self._generator:
  File "/home/mike/workspace/volatility3/volatility/framework/plugins/windows/malfind.py", line 108, in _generator
    for vad, data in self.list_injections(self.context, self.config["nt_symbols"], proc):
  File "/home/mike/workspace/volatility3/volatility/framework/plugins/windows/malfind.py", line 83, in list_injections
    for vad in proc.get_vad_root().traverse():
  File "/home/mike/workspace/volatility3/volatility/framework/symbols/windows/extensions/__init__.py", line 274, in traverse
    for vad_node in self.get_left_child().dereference().traverse(visited, depth + 1):
  File "/home/mike/workspace/volatility3/volatility/framework/symbols/windows/extensions/__init__.py", line 298, in get_left_child
    return self.Left
  File "/home/mike/workspace/volatility3/volatility/framework/objects/__init__.py", line 681, in __getattr__
    native_layer_name = self.vol.native_layer_name))
  File "/home/mike/workspace/volatility3/volatility/framework/objects/templates.py", line 72, in __call__
    return self.vol.object_class(context = context, object_info = object_info, **arguments)
  File "/home/mike/workspace/volatility3/volatility/framework/objects/__init__.py", line 113, in __new__
    value = cls._unmarshall(context, data_format, object_info)
  File "/home/mike/workspace/volatility3/volatility/framework/objects/__init__.py", line 284, in _unmarshall
    data = context.layers.read(object_info.layer_name, object_info.offset, length)
  File "/home/mike/workspace/volatility3/volatility/framework/interfaces/layers.py", line 492, in read
    return self[layer].read(offset, length, pad)
  File "/home/mike/workspace/volatility3/volatility/framework/layers/linear.py", line 37, in read
    for (offset, mapped_offset, mapped_length, layer) in self.mapping(offset, length, ignore_errors = pad):
  File "/home/mike/workspace/volatility3/volatility/framework/layers/intel.py", line 195, in mapping
    chunk_offset, page_size, layer_name = self._translate(offset)
  File "/home/mike/workspace/volatility3/volatility/framework/layers/intel.py", line 317, in _translate
    return self._translate_swap(self, offset, self._bits_per_register // 2)
  File "/home/mike/workspace/volatility3/volatility/framework/layers/intel.py", line 295, in _translate_swap
    swap_offset = swap_offset)
volatility.framework.exceptions.SwappedInvalidAddressException

The error should probably get caught, but it means that a vad parsed at some point was paged out. We'll look into handling the error gracefully, so I'll leave this open for a bit whilst we figure out the best way to handle it. @iMHLv2 Any thoughts on whether we can/should do error catching on this? Since it's happening in the object's helper method, I don't want to quash the exception unilaterally without documentation in the method to say, or catching it higher up...

[ikelos]

After some additional investigation by the original reporter(s) it looks like this might be a compressed memory page, which is represented in windows as the last available swap file (usually index 2, but potentially up to 15), hence the swap exceptions turning up. There's no support for this yet in volatility 3, but given we've got some swap index machinery already in place we're a little bit of the way there.

[github-actions[bot]]

This issue is stale because it has been open for 200 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 60 days since being marked as stale.