Add better support for detouring at instruction addresses with DHooks

[chrb22]

In https://github.com/alliedmodders/sourcemod/issues/1956 I explain the problem that the DHook post-hook causes when detouring at an arbitrary instruction address.

These commits add the new "calling convention" CallConv_INSTRUCTION, which ensures the post-hook is never created. For good measure I set it so you must use ReturnType_Void and ThisPointer_Ignore when setting up the hook as anything else doesn't make much sense. I also made it impossible to use params that aren't custom registers as an instruction detour doesn't have any proper arguments.

[chrb22]

I added instructions on how to avoid overriding jump targets after the hooked instruction. The current documentation says 6 bytes are patched, but isn't it only 5: e9 xx xx xx xx?

// If you need to detour the same function in different plugins make sure to
// wildcard \x2a the first 6 bytes of the signature to accommodate for the patched
// jump introduced by the detour.

Edit: Okay I see the #define JMP_SIZE 6, but couldn't it be 5?

[peace-maker]

Sorry for the delay, life happened. That warning about two plugins detouring the same function doesn't apply anymore since https://github.com/alliedmodders/sourcemod/pull/1642 for sourcemod plugins/extensions using the gamedata ecosystem. So we can probably leave it out or specify it's only needed when using other VSP or MMS plugins to detour the same functions.

I've envisioned a new parameter to the hooksetup / Functions section allowing you to specify how many bytes to skip instead of only the one instruction. That would require some smarter logic to know when to skip the whole trampoline, but just being able to skip the one instruction is a good start. We can add support for more complex usecases when they pop up. (Was thinking of this inline hook, but I don't plan to port this to sourcepawn :P)

[chrb22]

I was mostly asking about why 6 bytes are used instead of 5.

I was also thinking that it might be possible to add a stack parameter type, so you would be able to define an instruction hook solely using a gameconf. So something like

"stack var 1"
{
    "type"      "float"
    "stack"     "-0x3c"
}

Could be implemented by adding a new entry Stack = -1024 to Register_t so the location would internally be stored in the custom_register variable with a value of Stack - 0x3c. Would also make it possible to hardcode the location in the plugin by just passing DHookRegister_Stack - 0x3c to the hook setup instead of a register. Haven't looked at converting that to a Dhooks param that can be used in the plugin yet, but maybe it's just modifying CDynamicHooksSourcePawn::GetParamStruct and CDynamicHooksSourcePawn::UpdateParamsFromStruct?

[peace-maker]

I don't know, public/asm/asm.h defines it as 5 as well. Maybe @Ayuto can chip in? Might just be an oversight.

[Ayuto]

Yes, 5 seems to be correct. Good catch! 6 would be required for a far jump, but DynamicHooks isn't using that.

[chrb22]

I found out that it is also necessary to store the FLAGS register, but that requires me to add new instructions to the assembler from the sourcepawn submodule. I haven't used git a whole lot, so I don't know what the best way to do that is.

[chrb22]

The stack offset idea was surprisingly easy to implement: https://github.com/chrb22/sourcemod/commit/416c5b81ed610b5edb0154099d14c9ee6482eedf. It's a bit weird to put stack offsets together with the register enums, but it was the simplest way I could think of.