ghidra
ghidra copied to clipboard
NationalSecurityAgency
Undefined pCode "segment" exception on x86 real mode
Describe the bug Trying to step through John S Fine's boot12.asm bootloader code. This is a floppy disk bootsector. After setting the segment registers (ss = cs = ds = es = 0x0000), sp = bp = 0x7c00, the code does "mov [bp+24h], dl" (saving the drive number to the correct location in the memory image of the boot record). However, the emulator throws an exception.
Sleigh userop 'segment' is not in the library ghidra.pcode.exec.ComposedPcodeUseropLibrary@3b7b745d
ghidra.pcode.exec.PcodeExecutionException: Sleigh userop 'segment' is not in the library ghidra.pcode.exec.ComposedPcodeUseropLibrary@3b7b745d
at ghidra.pcode.exec.PcodeExecutor.step(PcodeExecutor.java:275)
⋮
---------------------------------------------------
Build Date: 2023-Dec-22 0936 EST
Ghidra Version: 11.0
Java Home: /usr/lib/jvm/java-17-openjdk-amd64
JVM Version: Private Build 17.0.9
OS: Linux 6.5.0-14-generic amd64
I told it to show the pCode for this code, and apparently it isn't translating any indirect memory references correctly?
0000:7c49 88 56 24 MOV byte ptr [BP + 24h],DL
$U1100:2 = INT_ADD BP, 36:2
$U3400:4 = CALLOTHER "segment", SS, $U1100:2
$U3b80:1 = COPY DL
STORE ram($U3400:4), $U3b80:1
The first pCode instruction is correct, as is the third one. The last one looks reasonably fine, too. The problem pCode is that second one which does a CALLOTHER "segment". I'm not sure where this is coming from or why it's there. The fact that it's referencing SS seems reasonable, since [SS:BP+24h] would be more explicit than [BP+24h] what it's trying to dereference. Why isn't this defined?
This is not uncommon to see in real mode code for an x86, which I suspect is probably the best-supported processor on Ghidra, so it surprises me that this operation is not defined. If this is not a bug, or someone knows how to solve this, please let me know.
To Reproduce
- Assemble boot12.asm using nasm -fbin boot12.asm
- Import the binary into ghidra
- Open it with the emulation tool
- Create a new trace, starting at the first instruction
- Step it until you reach the "mov [bp+24h], dl" instruction about 5 steps in.
Expected behavior I expect the memory location to be dereferenced correctly and not an exception because of an undefined pCode operation.
Attachments boot12.asm
Environment Build Date: 2023-Dec-22 0936 EST Ghidra Version: 11.0 Java Home: /usr/lib/jvm/java-17-openjdk-amd64 JVM Version: Private Build 17.0.9 OS: Linux 6.5.0-14-generic amd64
Additional context Boot12.asm is known to work correctly on real x86 systems, as posted.
This also seems to be generated when a call instruction is attempted.
The pCode here is CALLOTHER "segment", SS, SP because we're pushing the return address onto the stack.
In x86-16-real.pspec, this seems relevant and at least the formula seems correct:
<segmentop space="ram" userop="segment" farpointer="yes">
<pcode>
<input name="base" size="2"/>
<input name="inner" size="2"/>
<output name="res" size="4"/>
<body><![CDATA[
res = (zext(base) << 4) + zext(inner);
]]></body>
</pcode>
<constresolve>
<register name="DS"/>
</constresolve>
</segmentop>
I have no clue why it doesn't like things, unless it's not even loading in the x86-16-real.pspec file or something. This is pretty much the only userop set up here.
So, you are correct. The emulator does not currently heed segmentop elements in the language specifications. I'd been reluctant to do that by default, since some of those do not accurately model the actual processor. They might just model it well enough for decompilation or emulation with certain assumptions. In your case, x86-16-real.pspec happens to accurately model the actual operation.
Consider x86-16.pspec (protected mode). In reality, the processor applies a lookup table, but that is not what our spec models. It models a hard-coded lookup table (using a similar bit-shifting operation). As long as the image is loaded with that assumed table in mind, it'll work. All this to say, the specified segmentop is not always appropriate, but I suppose it works well enough most of the time.
I've put an internal ticket in to heed the segmentop elements in the emulator. In the meantime, you can implement the segment userop using a GhidraScript:
import ghidra.app.plugin.core.debug.service.emulation.AbstractDebuggerPcodeEmulatorFactory;
import ghidra.app.plugin.core.debug.service.emulation.BytesDebuggerPcodeEmulator;
import ghidra.app.script.GhidraScript;
import ghidra.debug.api.emulation.DebuggerPcodeMachine;
import ghidra.debug.api.emulation.PcodeDebuggerAccess;
import ghidra.debug.flatapi.FlatDebuggerAPI;
import ghidra.pcode.exec.*;
import ghidra.pcode.exec.PcodeArithmetic.Purpose;
public class InstallCustomSegmentOpScript extends GhidraScript implements FlatDebuggerAPI {
public static class CustomSegmentUseropLibrary<T> extends AnnotatedPcodeUseropLibrary<T> {
@PcodeUserop
public T segment(@OpExecutor PcodeExecutor<T> executor, T base, T inner) {
PcodeArithmetic<T> arithmetic = executor.getArithmetic();
long lBase = arithmetic.toLong(base, Purpose.OTHER);
long lInner = arithmetic.toLong(inner, Purpose.OTHER);
long lRes = (lBase << 4) + lInner;
return arithmetic.fromConst(lRes, 4);
}
}
public static class CustomSegmentUseropEmulator extends BytesDebuggerPcodeEmulator {
public CustomSegmentUseropEmulator(PcodeDebuggerAccess access) {
super(access);
}
@Override
protected PcodeUseropLibrary<byte[]> createUseropLibrary() {
return super.createUseropLibrary().compose(new CustomSegmentUseropLibrary<>());
}
}
public static class CustomSegmentUseropEmulatorFactory
extends AbstractDebuggerPcodeEmulatorFactory {
@Override
public String getTitle() {
return "Emulator with x86-16 Real Mode segmentation";
}
@Override
public DebuggerPcodeMachine<?> create(PcodeDebuggerAccess access) {
return new CustomSegmentUseropEmulator(access);
}
}
@Override
protected void run() throws Exception {
getEmulationService().setEmulatorFactory(new CustomSegmentUseropEmulatorFactory());
}
}
You'll need to run this script each time you open the Debugger or Emulator tool. It'll remain active in that tool until you close it.
@nsadeveloper789 if I wanted to do the same for the x86-16.pspec (protected mode), would it be correct just to replace the line
long lRes = (lBase << 4) + lInner;
with
long lRes = (lBase << 16) + lInner;
or is that too simplistic??
@Wall-AF No, it would be more like using the segment as an index into an array, if 32b mode is anything to go by. The value returned for that particular index (or "selector" in 32bit protected parlance) would be the "base". The CPU also looks at that array entry and checks if your memory reference is in-bounds, you have privs to access it, etc, and if one of those checks fail, you get the dreaded segfault. In real mode, there's no special struct to be dereferenced, that "base" is just a group ("segment") of 64kb chunks of memory. There's no setup to be done for it, this is just how the CPU handles memory (it's a hack to the architecture so the 8086+ can address more than 64k of memory) If you're interested in learning how this stuff works, OSDEV has some really great information. FYI, that array I referred to earlier? That's normally set by the operating system and the ring 3 programs (user/unprivileged mode, what your program runs in) never get to mess with it, they're not even allowed to change the selectors!
@nsadeveloper789 I appreciate your thoughtful response. I will try this out tonight :) Thank you.
@Wall-AF In general @Hordeking is correct. The provided x86-16.pspec provides that implementation as a sort of hack. It's good enough for decompilation. If you carefully load your target image at the appropriate addresses, then that hack may work for you in emulation as well. Alternatively, you can hard-code a table in the Java callback, rather than dereferencing an actual in-target-memory structure. Your image and that hard-coded table have to be consistent, though. Each segment must be imported at the expected address.
@Wall-AF Keeping that stuff consistent sounds like a positive pain in the butt. You should implement the IDT and GDT stuff :grinning:
@nsadeveloper789 I tried it. It works superbly. It's good enough that I can fake the int 13 calls to load stuff and check that my numbers from walking through the code match it's actually doing in there. Thanks!
@nsadeveloper789 So, should I go ahead and close this as resolved? I'm assuming an internal ticket probably means it'll show up in the next couple of public releases at this point? Or I can leave it open, or if you would rather close it with a tag for others to fine, I'm cool with that.
The internal ticket merely indicates my intent. It's still pending our prioritization process, so I would not take that to mean it'll show up in the next couple of public releases. Please leave the issue open. We'll close it when we've either completed or rejected the internal ticket. My best guess is we'll accept it, but it'll wind up pretty far back in my queue.