ghidra icon indicating copy to clipboard operation
ghidra copied to clipboard

Ability to have more than `Byte.MAX_VALUE` calling conventions

Open lab313ru opened this issue 1 year ago • 3 comments

Subj. This can be used to generate custom calling conventions with different input and output destinations. For now you cannot apply more than 127 calling conventions using HighFunctionDBUtil.writeOverride, it prints

ERROR Unable to assign calling convention `__r39_r56` - allocation capacity exceeded (CallingConventionDBAdapterV0)  

lab313ru avatar May 16 '23 01:05 lab313ru

That seems rather excessive and is likely abusing the intention of reuseable "named" calling conventions. I assume this is being done to customize storage on a function-by-function basis or do you intended on reusing each named convention to multiple functions? Our intent is to add custom storage capability to Function Definitions in a manner similar to that of Function.

ghidra1 avatar May 22 '23 15:05 ghidra1

I am reaching a similar issue here reverse engineering hand-generated code; so without a consistent calling "convention".

The information from the "calling convention" I do not see in the custom storage is the list of affected (and symmetrically, unaffected) registers; from which the true output registers can be drawn (using the custom storage).

A large number of calling convention allows for a fine-grained analysis, without which I would be missing the complete effect of the function (returning a structure across multiple registers) or on the contrary been swamped with non-existant side-effects.

But in my case being able to set the killed by / unaffected by lists on a function by function basis would also solve my issue.

jibee avatar Apr 25 '24 17:04 jibee

For taylored situations as you describe (e.g., pure assembly code) we are evaluating an approach which would allow a function-specific tayloring of affected/unaffected registers, etc.. We are also evaluating similar storage-specific details for function definitions (i.e., data type). We prefer not to blow-out the number of defined calling conventions used at this point in time.

ghidra1 avatar May 02 '24 17:05 ghidra1