PaX RAP optimization summary and ending

[citypw]

Hi guys,

We've been doing this project from time to time in past two years. The most work is done by @fanfuqiang [a.k.a "zet"] . Our initial target was to figure out how PaX RAP work and the possibility for performance optimization. We did some modification to speed up the forward-edged CFI after analysis on how RAP being implemented as a GCC plugin and the potential issues what PaX RAP may have. zet documented( it's a Chinese version, please use Google translate) it about the findings he found:

• 1. RAP didn't use pointer analysis leds to output all hashes for all functions
• 2. some code inserted( instrumentation?) by RAP after GCC tree-ssa optimiation so it won't be optimized by GCC itself
• 3. the same func prototype would only generate the same hashes could be exploited in some ways?

I was a bit of confused by 3) IIRC this potential issue was also mentioned by the better-than-none solution is called kCFI resulted in the attacker is unlikely to exploit it in the real production. I wasn't sure so I confirmed it with PaX team which it is by design originally. PaX team also pointed out:

• Our claim about PaX RAP algorithm is to insert the type hash before every function is not true. The type hash will be prepended only to those functions which can be a target of indirect calls. In practice that means that they're either public (so other translation units can take their address, compiling under LTO will help this case) or their address has been taken (in the current TU) for all other functions a hash value of 0 is inserted which is a special reserved value that no type can hash to. What possibly optimization can do here over GCC's existing analysis is the question we can't answer now.

• Only the first part could be improved but that requires building with LTO( not implemented in the public patch) and then GCC will do a proper job already. when you're stuck with a per-TU compilation model, then you can't say anything about all non-static functions, the type hash must be emitted for them, there's nothing to improve there(or rather, you can still improve even in this case but that doesn't require any further points-to analysis).

• To 1), it's not how rap works, not all functions get the type hash. To 2), from PaX team's opinion not sure what is there to optimize? To 3), it's a type hash based target categorization, it's by design

• Our modification did not emitting a 0 hash, that's bad because any previous bytes of such functions will serve as the hash so there's a chance that they will match an actual type hash and make this function available for attacks (which an explicit 0 hash prevents).

• The real hard problem is the non-type based categorization

I really appreciate for PaX team's feedback that does makes more sense to me. I've been getting involved with the solution with PaX/Grsecurity for a very long time. I know some of their engineering criteria and threat model are usually more rigorous than the average case. That's why I was confusing about my misunderstanding about the design and implementation in the 1st place. This is only an experimental project. In the end, I agree with"any 'optimization' shouldn't be achieved is by eliminating instrumentation which means reducing security, especially not a wise trade-off for a security feature. We failed to see through the trade-off in the original design. Now we learned;-)

As the big changes in the current PaX RAP which becomes more powerful with complexity since the arrival of MELTDOWN/SPECTURE variants changed a lot of security measures. This project no longer need to be continued because IMOHO HardenedLinux's goal should be the possible solution for the production. I'd say RAP-optimizations ain't fit my initial ticks from now on.

Anyway, thanks for zet's hard work in past two years. Thanks for PaX team's review( this assessment is what I really need desperately). Thanks to all contributors of PaX/GRsecurity in past 18 years.