vimeo
Option to disable JIT for better performance
I'm on Windows 11, with Psalm 6.13.1 and PHP 8.4.14 (x64 NTS build).
Every time I run Psalm, it displays "JIT compilation in progress..." for about 5 seconds before starting to scan files. There doesn't seem to be a configuration option to turn this off, but for testing I manually edited vendor/vimeo/psalm/src/Psalm/Internal/Fork/PsalmRestarter.php, setting 'enable_cli' => 0 in the REQUIRED_OPCACHE_SETTINGS array.
After doing this, I see a consistent 5 second reduction in the time it takes to run Psalm on my project. I benchmarked 10 runs before and after the above change (using time psalm in Bash). While memory usage increased by about 10%, the total time to run Psalm is consistently 33% faster:
Perhaps there is a benefit to always enabling JIT compilation on other platforms (the docs mention an up to +20% performance increase), but at least on Windows it would be nice to have an option to disable it for better performance.
@kkmuffme No, this is not related to Psalm restarting the process, since in my test it is being restarted in both cases (just with opcache.enable_cli=0 in the "JIT OFF" runs).
The main thing that can have some impact is preloading in combination with JIT.
On smaller codebases preloading has a negative performance impact, on an medimum-size codebases it has a positive impact.
JIT by itself without preloading has pretty much the same behavior, only the positive impact starts earlier (medium-small codebases).
I am currently working on a PHP AOT compiler (mainly for Psalm), which should solve the issue altogether.
If anything, there should be a flag to control preloading, however preloading fixes a negative performance impact caused by repeated JIT compilation in multithreaded mode.
A possible optimization path can be the reduction of the number of preloaded classes (arguably there are some classes which probably shouldn't be in the preloaded list as they aren't even used in multithreaded mode, like the LangaugeServerProtocol classes, AdvancedJsonRpc, LibDNS).
It might actually make more sense to just disable preloading (without touching JIT) when running in singlethreaded mode, that would speed up execution without slowing down multithreaded mode.
I am currently working on a PHP AOT compiler (mainly for Psalm), which should solve the issue altogether.
I'm not sure if this is time well spent. There are recurring discussions about that since probably a decade, however with the current level of JIT the performance benefit is minimal - see https://github.com/php/php-src/issues/8203#issuecomment-1070905991 - at least in terms of PHP performance/PHPs working model
I tested disabling preloading (by commenting out the Preloader::preload() call at the end of Psalm::restart()). This does have a benefit, but not nearly as much as disabling the forced JIT as PR #11613 does.
Test results (average of 10 runs):
| Configuration | Real time (seconds) | Improvement |
|---|---|---|
| JIT + preloading (current Psalm) | 15.6 | 0% |
| JIT with preloading disabled | 13.5 | 13% |
| JIT off + preloading (PR #11613) | 10.3 | 34% |
| JIT off, preloading disabled | 10.0 | 36% |
I still would like to have an option to disable Psalm's forced overriding of Opcache settings, as this has a much bigger benefit to me than disabling preloading.
What platform, CPU and PHP version are you running this on? This info can be useful as your results don't quite match mine, and it might be useful to disable preloading/JIT on specific platform configs
I get the following results when scanning Psalm itself (average of 6 runs, +-300ms jitter between runs):
| Configuration | Real time (seconds) | Improvement |
|---|---|---|
| JIT + preloading (current Psalm) | 50.5 | 0% |
| JIT with preloading disabled | 49.3 | 2% |
| JIT off + preloading (PR #11613) | 46.9 | 7% |
| JIT off, preloading disabled | 46.8 | 7% |
I did the same test scanning Psalm itself on a different Windows 11 laptop, this one with a 13th gen Core i9-13900HX CPU and PHP 8.5.0 x64 NTS. Again I averaged 6 runs for each configuration, and observed jitter of up to +-300ms between runs. Results:
| Configuration | Time (seconds) | Improvement |
|---|---|---|
| JIT + preloading (current Psalm) | 44.1 | 0% |
| JIT with preloading disabled | 43.2 | 2% |
| JIT off, preloading disabled | 42.0 | 5% |
| JIT off + preloading (PR #11613) | 41.5 | 6% |
Interestingly, on this PC having preloading enabled seems to help performance slightly when the JIT is off (but not when it is on).
To compare performance on Linux, I installed Ubuntu 24.04 + PHP 8.5.0 on the above laptop (13th gen Core i9-13900HX CPU). By default Psalm scanned itself in 13 seconds using 32 threads, but I was able to get slightly better performance (11.6 seconds) by running with --threads=8 --scan-threads=8 (there are 8 performance cores on this CPU). I then compared the same 4 configurations as on Windows, all using 8 threads:
| Configuration | Time (seconds) | Improvement |
|---|---|---|
| JIT + preloading (current Psalm) | 11.6 | 0% |
| JIT with preloading disabled | 10.8 | 7% |
| JIT off + preloading (PR #11613) | 10.5 | 9.5% |
| JIT off, preloading disabled | 10.5 | 9.5% |
I'm curious to know what project or platform actually benefits from the JIT, since for the codebases I've been able to test on both Windows and Linux the JIT as configured by Psalm hurts rather than helps performance.
My results match as well (average of 5 runs each on 8.5.0, arch linux, 12th Gen Intel(R) Core(TM) i5-12450H):
array(4) {
["jit + preload"]=>
float(12.847333333333333)
["jit only"]=>
float(11.669166666666667)
["no jit no preload"]=>
float(10.592500000000001)
["no jit preload"]=>
float(10.251)
}
These results actually do not match the results I got when I initially added the feature: if I remember properly, this might actually be related to a bugfix within the JIT (after a bugreport I submitted) which increased compilation times, either that or it was one of the recently merged JIT refactors.
Either way, looks like JIT is not viable anymore even for small-medium sized projects like Psalm itself (will check with the larger codebases at one of my clients tomorrow). One more reason to quickly finish my AOT compiler ;)
In the meantime, after I get the results on the larger codebase tomorrow, I'll likely submit a PR with an option to disable JIT and preloading separately (with a tip to enable it if the scan runs for too long).