plutus icon indicating copy to clipboard operation
plutus copied to clipboard

Some benchmarks are screwed

Open effectfully opened this issue 1 year ago • 1 comments

#5867 fixed some of the screwed up benchmarks (See this comment for an explanation of what went wrong), but we still have more of these issues in

  • budgeting-bench
  • marlowe
  • bls12-381-costs
  • potentially more

These need to be fixed as well. It would also be great to try to ensure we don't run into this issue again, although I'm not sure if it's possible.

effectfully avatar Apr 08 '24 20:04 effectfully

I did this for bls12-381-benchmarks, marlowe, and cek-calibration in #5905. However

  • The Marlowe benchmarks showed a speedup of 10-20%. It might be instructive to find out why this is. The speedup seems to be inversely correlated with the size of the script: the largest scripts have the smallest speedup. The relationship seems quite weak though, and we don't have a big enough selection of different script sizes to come to a firm conclusion.
  • The validation benchmarks initially slowed down by 2%, despite the fact that they weren't changed. This seems to depend on the order and number of exports from PlutusBenchmark.Common: see #5906. Trial and error led to an arrangement of the exports that got the validation times back to where they were, but that's not a reliable way of doing things.

kwxm avatar Apr 16 '24 00:04 kwxm