benchexec
benchexec copied to clipboard
sosy-lab
Randomize the size of the environment for started processes
Mytkowicz et al. found that adding a random number of bytes to the environment has an important effect on the end runtime because it affects memory layout, which could align or disalign with cache blocks. Should benchexec contain an option like --randomize-env/--no-randomize-env and --randomize-env-max-padding <max size>, which inserts a uniformly selected random number of bytes to the environment?
I am willing to implement this feature if it makes sense in benchexec.
This is related to #13. We are aware of the issue with environment variables, but it is not easy to decide what to do.
Randomized padding would make sense only if one repeats all measurements several times and takes the average. This is something that many users of BenchExec could never do (due to how much time it takes to just execute all benchmark runs once) and I am wondering how many users of BenchExec exist that do execute their benchmark runs several times, because so far BenchExec does not provide any special features for this case and there has been a complete lack of feature requests in this regard (cf. #19). Do you run everything several times?
In general, the approach taken by BenchExec is to control external factors where possible instead of randomizing them. This would mean implementing #13 and restricting the environment variables of the executed process to a minimal set of environment variables that ideally have constant values. The reason why this was not done yet is that it is hard to know what the set of environment variables should be. For example, PATH is a specific problem, we probably can not hard-code it to a fixed value.
Given the result from Mytkowicz, I don't think it's possible to make a performance comparison based on a single run. But that of course is a philosophical question for the users to decide.
Controlling random factors that isolate noise makes sense when the controlled factors should model performance in a practical setting. Cutting off the network falls into this category; even though real systems will probably have a network, the performance of network-disabled systems should model the performance of network-enabled systems plus some random factor that can be modeled independently.
However, if you run a benchmark with env size of K, you have no guarantee that this models the performance of that benchmark env size of K + 1. There is no single value that will be statistically representative of real world circumstances. The only way to isolate the impact of the environment size is to run multiple times with different env sizes and bootstrap statistics over those runs; I don't think selecting a single deterministic env size (as in #13) will isolate the effect of the env size. Of course, fixing the environment variables is still useful for reproducibility reasons, and it should probably be done whether or not one chooses to insert a randomly sized variable.
As you pointed out, some users will not be able to run multiple times, so they will just have to live with the knowledge that their measurements have an additional ~5% uncertainty that Mytkowicz found. If these users are more common, then env randomization should be off by default. But I think it should be an option for the case where multiple runs are possible.
Sure. If there are actually users who would benefit from this feature it makes sense to add it.
Having it off by default and enabled with an argument like --randomize-env-size sounds good. What would be a good maximum size? I have not thought much about it, but my first idea would be the size of one memory page. Do we need an option for changing the maximum size, what would the use case be?
One page (getconf PAGESIZE) probably makes sense as a maximum; no need to provide a flag to change that.
Hi, wondering if there is any data showing the effect of randomizing the size of environment variables on programs running on modern CPUs? The paper is more than 10 years old and there are a lot of improvements to CPUs in the past 10 years, such as better hardware prefetcher and much larger caches. Wondering if the effect is still that large as mentioned in the paper.
I am not aware of any more recent published results, but I also haven't looked for such results recently.