py-evm icon indicating copy to clipboard operation
py-evm copied to clipboard

Published 20 hours ago • verified publisher icon ethereum

[WIP] Add Stack and Computation Benchmark

Open Bhargavasomu opened this issue 6 years ago • 4 comments

What was wrong?

As part of the discussion here, a new Stack Benchmark is needed.

How was it fixed?

A TestStack.sol contract was written which would serve as the benchmark.

Cute Animal Picture

put a cute animal picture link inside the parentheses

Bhargavasomu avatar Jan 09 '19 17:01 Bhargavasomu

@pipermerriam I still haven't made this a benchmark as of yet, but this is my first time where I am running a contract on the EVM, and I just had a small doubt.

  • For the above TestStack Contract which you mentioned, I ran that code through our EVM, but the final stack values that I am seeing is [2729995958]. As per the above code shouldn't the expected value be [10,000].
  • If at all this is stored in Memory (which is in bytes), how could I obtain the final value of v, because I would like to put an assert statement there.

Or is there anything else I am missing? Could you please clarify? Thankyou

Bhargavasomu avatar Jan 09 '19 17:01 Bhargavasomu

Also you can find the above stated values in py36-stack-benchmark

Bhargavasomu avatar Jan 09 '19 17:01 Bhargavasomu

@pipermerriam I am extremely sorry, I think I messed up with git and it is showing some of the previous commits as mine. Will fix it immediately. The Merkle Tree related things aren't supposed to be part of this PR.

Bhargavasomu avatar Jan 09 '19 18:01 Bhargavasomu

For the benchmarking of this I think the final value doesn't matter very much. It's more about determining how many times the stack is pushed/popped from (and maybe even in what quantities).

I only just now realized that https://github.com/ethereum/py-evm/pull/1515 still hasn't been completed as it would provide an ideal API for this. Without that I think you're limited to manually measuring it using something like unittest.mock and then just hard coding the numbers in the benchmark.

Then, suppose the contract code results in 150 stack pushes and 150 stack pops, then you can output some metrics like N pops / second. You should probably run the computation 100 times or something like that to get a sufficiently large sample size.

Then later when #1515 is merged we can adjust the code to dynamically measure the number of push/pops.

pipermerriam avatar Jan 09 '19 18:01 pipermerriam