java-smt icon indicating copy to clipboard operation
java-smt copied to clipboard

Use the partial model for formula evaluation in Princess

Open daniel-raffler opened this issue 10 months ago • 2 comments

Hello, We stopped using the partial model in Princess here as there were several solver issues when using it with rational formulas. Most of these issues have now been resolved by the Princess developers and we should look into whether it's possible to return to using the partial model in JavaSMT.

daniel-raffler avatar Jan 11 '25 11:01 daniel-raffler

There is an issue in Ostrich/Princess that affects String evaluation and is blocking us from using the partial model for now. I've opened an issue here: https://github.com/uuverifiers/ostrich/issues/94

daniel-raffler avatar Jan 16 '25 23:01 daniel-raffler

Other than that it seems to be working fine. I've updated the Princess backend and ran some benchmarks with CPAchecker:

Image

The timeout is 60 seconds and --option cpa.predicate.encodeFloatAs=RATIONAL was used to specifically test rational formulas as they used to cause crashes when being evaluated. The results on the left are with the current version, and the new version that uses the partial model can be found on the right. The old version produces a large number of incorrect false results that are avoided with the new code. However, these runs will still result in an Assertion error with the new version, and the message reads "Found imprecise counterexample with BMC".

The new version also returns fewer correct results, and performance seems to be slower, especially for more expensive runs:

Image

Here the blue/green line is the old version, and the brown line is with the new version that uses the partial model. I would suggest we still update as soon as the bug is fixed as this will allow us to remove the current workaround for the partial model from our code.

daniel-raffler avatar Jan 16 '25 23:01 daniel-raffler