volatilityfoundation
Feature/better page faulting
This shifts the transition page handling into the windows fault handler.
It makes the table traversal from a loop into a recursive call so that handlers can jump back into the traversal directly.
This supersedes #518. It should have the same effect (translation is only carried out if there the entry is present).
It should also allow for the fault handler to be called multiple times.
the unknown_bit should probably be removed. it is part of the protection field, so checking for it means you only handle pages with a certain protection: https://github.com/volatilityfoundation/volatility3/blob/44d8c0d93139f37f0b0725d72942f0703a7e4dc8/volatility3/framework/layers/intel.py#L324
https://github.com/volatilityfoundation/volatility3/blob/44d8c0d93139f37f0b0725d72942f0703a7e4dc8/volatility3/framework/layers/intel.py#L347
furhtermore, i think there is a bug in the translation process.
with a previous version of vol3 i get this for a swapped page:
proc_layer.translate(0x26eb0dde000)
(48635904, 'swap_layers0')
using the code from this PR, i get this:
proc_layer.translate(0x26eb0dde000)
PagedInvalidAddressException Traceback (most recent call last)
...
volatility3/framework/layers/intel.py in _translate(self, offset)
113 # Now we're done
114 if not (entry & 0x01):
--> 115 raise exceptions.PagedInvalidAddressException(self.name, offset, position + 1, entry,
...
Forgot to add this:
I recognized that Windows changed the _MMPTE_SOFTWARE struct (amongst others) during the last Windows versions, leading at least to an incorrect parsing of the pagefile index (PageFileLow) , since the offset is hardcoded: https://github.com/volatilityfoundation/volatility3/blob/develop/volatility3/framework/layers/intel.py#L332
So all hardcoded offsets related to MMPTE structs should be gathered dynamically from the profile.
Also, with newer Windows versions (based on basic tests, the earliest version is Windows 10 1803 17134; Windows 10 1709 16299 still behaves "normally"), there seems to be a new (potential) bitflag, which is undocumented so far. This flag is part of the PageFileHigh field and hence leads to a wrong swap/pagefile offset. Using !pte to resolve the pagefile offset shows, that !pte simply ignores/unsets this bit to calculate the offset. It should be noted, however, that I didn't observe this to be unset so far. This flag seems also to be set for demand zero PTEs. Due to this flag, they appear to have a PageFileHigh value of 0x2000 (only bit 13 set), resp. a swap/pagefile offset of 0x2000000, but resolving those with WinDbg !pte returns a demand zero PTE.
My current best guess is to simply unset it for affected Windows versions. This should, however, be investigated further.
@f-block @ikelos if you provide me with the said dump I will investigate this and make this PR happen
Hi,
thanks for the efforts @paulkermann, but it doesn't quite solve the problems.
To reproduce the following commands, you can use this dump: https://fx.ernw.de/portal-seefx/~public/YzMxMzkxZDAtY2I0My00M2Q5LWIzZjMtNGIwYTcyNjIwMTgz?download
When trying e.g., to resolve a virtual address for a swapped page, while providing a pagefile, I get this error:
(layer_name_Process964_Process964) >>> context.layers['layer_name_Process964_Process964'].translate(0x19ca37f7000)
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/opt/forensic_volatility3-git/volatility3/framework/layers/linear.py", line 14, in translate
mapping = list(self.mapping(offset, 0, ignore_errors))
File "/opt/forensic_volatility3-git/volatility3/framework/layers/intel.py", line 218, in mapping
for offset, size, mapped_offset, mapped_size, map_layer in self._mapping(offset, length, ignore_errors):
File "/opt/forensic_volatility3-git/volatility3/framework/layers/intel.py", line 251, in _mapping
mapped_offset, _, layer_name = self._translate(offset)
File "/opt/forensic_volatility3-git/volatility3/framework/layers/intel.py", line 120, in _translate
raise exceptions.PagedInvalidAddressException(self.name, offset, position + 1, entry,
volatility3.framework.exceptions.PagedInvalidAddressException: Page Fault at entry 0xed1f00000082 in page entry
Checking this PTE with my plugin (see here) shows that it is a PTE in SOFTWARE state which belongs to a pagefile and has a valid offset into the pagefile:
vol.py -f mem.dump windows.pte_resolve --pid 964 --vaddr 0x19ca37f7000
...
PteRun:
PID: 964
vaddr: 0x19ca37f7000
swap_offset: 0xcd1f000
pagefile_idx: 0
length: 0x1000
pte_value: 0xed1f00000082
...
MMPTE struct:
=============
[_MMPTE_SOFTWARE] @ 0x253b4fb8
0x0 ColdPage 0x0 bitfield (bit 3)
0x0 PageFileAllocated 0x0 bitfield (bit 2)
0x0 PageFileHigh 0xed1f bitfield (bits 32-64)
0x0 PageFileLow 0x0 bitfield (bits 12-16)
0x0 PageFileReserved 0x1 bitfield (bit 1)
0x0 Protection 0x4 bitfield (bits 5-10)
0x0 Prototype 0x0 bitfield (bit 10)
0x0 ShadowStack 0x0 bitfield (bit 26)
0x0 SwizzleBit 0x0 bitfield (bit 4)
0x0 Transition 0x0 bitfield (bit 11)
0x0 Unused 0x0 bitfield (bits 27-32)
0x0 UsedPageTableEntries 0x0 bitfield (bits 16-26)
0x0 Valid 0x0 bitfield (bit 0)
Another still existing problem: The dynamic field-offset generation (using the currently loaded profile) must be implemented somewhere (see here). As can be seen in the _MMPTE_SOFTWARE struct, the PageFileLow field is for newer Windows versions not at offset one anymore but at offset 12. This leads to a wrong n value (here 1 instead of 0).
The way I'm doing this in my plugin can be seen here. But I'm not sure if that is the intended way to do it. Especially because the kernel/structures don't seem to be available at any given point within the intel classes. If somebody gives me an example on the best way to access the structures here, I could add code for this.
Also: As commented here, the unknown_bit must be removed as it tests a bit in the SOFTWARE Protection and hence only matches for certain protections. I rechecked this particular case: This test prevents in particular the translation of pages with MM_READONLY (0x1), MM_EXECUTE (0x2) and MM_EXECUTE_READ (0x3):
Here a SOFTWARE PTE with a MM_EXECUTE_READ protection:
vol.py -f mem.dump windows.pte_resolve --pid 964 --vaddr 0x7df428831000
...
PteRun:
PID: 964
vaddr: 0x7df428831000
swap_offset: 0x152bb000
pagefile_idx: 2
length: 0x1000
pte_value: 0x172bb00002074
...
[_MMPTE_SOFTWARE] @ 0x380ea188
0x0 ColdPage 0x0 bitfield (bit 3)
0x0 PageFileAllocated 0x1 bitfield (bit 2)
0x0 PageFileHigh 0x172bb bitfield (bits 32-64)
0x0 PageFileLow 0x2 bitfield (bits 12-16)
0x0 PageFileReserved 0x0 bitfield (bit 1)
0x0 Protection 0x3 bitfield (bits 5-10)
0x0 Prototype 0x0 bitfield (bit 10)
0x0 ShadowStack 0x0 bitfield (bit 26)
0x0 SwizzleBit 0x1 bitfield (bit 4)
0x0 Transition 0x0 bitfield (bit 11)
0x0 Unused 0x0 bitfield (bits 27-32)
0x0 UsedPageTableEntries 0x0 bitfield (bits 16-26)
0x0 Valid 0x0 bitfield (bit 0)
Its PTE value ("entry" value) is 0x172bb00002074. Using the unknown_bit test results in 0 and would prevent a translation:
0x172bb00002074 & (1<<7) = 0
I'm guessing the original reason to use this bit was that there is no specific SOFTWARE bit and this bit was typically set for most paged out pages (there are not that many swapped pages without that bit, but there are some!). This is, however, obviously not a good test and should be removed. The way I test for this case is similar to Volatility's second if condition (entry >> self._swap_bit_offset) != 0), which essentially checks for the PageFileHigh field. From my observations, this check should be fine under one condition: Bit 13 must be unset (see here), otherwise SOFTWARE PTEs, not corresponding to any swapped page, will be misinterpreted.
I could add some code to unset this bit for the appropriate Windows versions (similar to here), but would wait until feedback regarding the dynamic-offset generation.
Cheers, Frank
@f-block
I agree with you about the whole dynamic-offset generation thing and I think that this information should be available to the layer.
Also, it looks like if the swizzle bit is not set then the whole PTE value is anded with a mask (from MiInvalidPteMask or the MiState).
This way the PageFileHigh value seems to give the real swap file offset.
Also, I don't understand your problem with the Protection field. Value of 3 means MM_EXECUTE_READ and indeed in your PTE it shows the value of 3. I will remove the whole unknown_bit logic thingy.
I will try to create a new automagic to attach the main kernel module to the layer so structure offsets would be available in the page fault handler.
Hi @paulkermann,
thanks for the hint regarding the swizzle bit and the MiInvalidPteMask, I will have a look if that correlates with my observations.
Regarding the Protection field: Sorry, I probably should have explained this a bit more detailed: The unknown_bit checks for the 2nd bit (counting from zero) in the Protection field (resp. the 7th bit for the whole entry), that means that every protection not having this bit set, which is e.g., the case for the protection values 0-3 (0b00 - 0b11), will be ignored, but shouldn't be!
Looking forward for the new automagic functionality.
Cheers, Frank