Create blockchains that consume full gas on single operations.

[marioevz]

Requirement for zkevm testing:

Create a test that exhausts the gas stipend of a full block using a single operation, operation meaning:

• An opcode, namely keccak or maybe other relevant ones.
• A precompile

Potentially also parametrizing whether the tx succeeds or fails (reverts or out-of-gas).

cc'ing @kevaundray to expand or correct the definition of operations

[raxhvl]

Looking into this. Planning to extend ethereum/execution-spec-tests#1456 by @jsign

[kevaundray]

EVM opcodes

Arithmetic

• [x] 0x01 ADD ethereum/execution-spec-tests#1571
• [x] 0x02 MUL ethereum/execution-spec-tests#1571
• [x] 0x03 SUB ethereum/execution-spec-tests#1571
• [x] 0x04 DIV ethereum/execution-spec-tests#1571
• [x] 0x05 SDIV ethereum/execution-spec-tests#1571
• [x] 0x06 MOD ethereum/execution-spec-tests#1571
• [x] 0x07 SMOD ethereum/execution-spec-tests#1571
• [x] 0x08 ADDMOD ethereum/execution-spec-tests#1571
• [x] 0x09 MULMOD ethereum/execution-spec-tests#1571
• [x] 0x0A EXP ethereum/execution-spec-tests#1571
• [x] 0x0B SIGNEXTEND ethereum/execution-spec-tests#1571
• [x] 0x10 LT ethereum/execution-spec-tests#1571
• [x] 0x11 GT ethereum/execution-spec-tests#1571
• [x] 0x12 SLT ethereum/execution-spec-tests#1571
• [x] 0x13 SGT ethereum/execution-spec-tests#1571
• [x] 0x14 EQ ethereum/execution-spec-tests#1571
• [x] 0x15 ISZERO ethereum/execution-spec-tests#1571
• [x] 0x16 AND ethereum/execution-spec-tests#1571
• [x] 0x17 OR ethereum/execution-spec-tests#1571
• [x] 0x18 XOR ethereum/execution-spec-tests#1571
• [x] 0x19 NOT ethereum/execution-spec-tests#1571
• [x] 0x1A BYTE ethereum/execution-spec-tests#1571
• [x] 0x1B SHL ethereum/execution-spec-tests#1571
• [x] 0x1C SHR ethereum/execution-spec-tests#1571
• [x] 0x1D SAR ethereum/execution-spec-tests#1571

Cryptographic

• [x] 0x20 KECCAK256 ethereum/execution-spec-tests#1497

Stateful Opcodes

• [x] 0x31 BALANCE ethereum/execution-spec-tests#1583
• [x] 0x3B EXTCODESIZE ethereum/execution-spec-tests#1583
• [x] 0x3C EXTCODECOPY ethereum/execution-spec-tests#1583
• [x] 0x3F EXTCODEHASH ethereum/execution-spec-tests#1583
• [x] 0x40 BLOCKHASH ethereum/execution-spec-tests#1583
• [x] 0x47 SELFBALANCE ethereum/execution-spec-tests#1583
• [x] 0x54 SLOAD ethereum/execution-spec-tests#1583
• [x] 0x55 SSTORE ethereum/execution-spec-tests#1583
• [x] 0xF1 CALL ethereum/execution-spec-tests#1583
• [x] 0xF2 CALLCODE ethereum/execution-spec-tests#1583
• [x] 0xF4 DELEGATECALL ethereum/execution-spec-tests#1583
• [x] 0xFA STATICCALL ethereum/execution-spec-tests#1583

Termination

• [x] 0x00 STOP (Rationale: Cannot fill a block with STOP opcodes)
• [x] 0xFE INVALID (Rationale: Cannot fill a block with INVALID opcodes)

Environment

• [x] 0x30 ADDRESS ethereum/execution-spec-tests#1651
• [x] 0x32 ORIGIN ethereum/execution-spec-tests#1651
• [x] 0x33 CALLER ethereum/execution-spec-tests#1651
• [x] 0x34 CALLVALUE ethereum/execution-spec-tests#1651
• [x] 0x35 CALLDATALOAD ethereum/execution-spec-tests#1657
• [x] 0x36 CALLDATASIZE ethereum/execution-spec-tests#1651
• [x] 0x37 CALLDATACOPY ethereum/execution-spec-tests#1690
• [x] 0x38 CODESIZE ethereum/execution-spec-tests#1651
• [x] 0x39 CODECOPY ethereum/execution-spec-tests#1690
• [x] 0x3A GASPRICE ethereum/execution-spec-tests#1651
• [x] 0x3D RETURNDATASIZE ethereum/execution-spec-tests#1651
• [x] 0x3E RETURNDATACOPY ethereum/execution-spec-tests#1690

Block & Chain Info

• [x] 0x41 COINBASE ethereum/execution-spec-tests#1651
• [x] 0x42 TIMESTAMP ethereum/execution-spec-tests#1651
• [x] 0x43 NUMBER ethereum/execution-spec-tests#1651
• [x] 0x44 PREVRANDAO ethereum/execution-spec-tests#1651
• [x] 0x45 GASLIMIT ethereum/execution-spec-tests#1651
• [x] 0x46 CHAINID ethereum/execution-spec-tests#1651
• [x] 0x48 BASEFEE ethereum/execution-spec-tests#1651

Blobs

• [x] 0x49 BLOBHASH ethereum/execution-spec-tests#1657
• [x] 0x4A BLOBBASEFEE ethereum/execution-spec-tests#1651

Stack & Memory

• [x] 0x50 POP ethereum/execution-spec-tests#1657
• [x] 0x51 MLOAD ethereum/execution-spec-tests#1657
• [x] 0x52 MSTORE ethereum/execution-spec-tests#1668
• [x] 0x53 MSTORE8 ethereum/execution-spec-tests#1668
• [x] 0x59 MSIZE ethereum/execution-spec-tests#1651
• [x] 0x5A GAS ethereum/execution-spec-tests#1651
• [x] 0x5C TLOAD ethereum/execution-spec-tests#1657
• [x] 0x5D TSTORE ethereum/execution-spec-tests#1657
• [x] 0x5E MCOPY ethereum/execution-spec-tests#1690
• [x] 0x5F PUSH0 ethereum/execution-spec-tests#1651
• [x] 0x60-0x7F PUSH1-PUSH32 ethereum/execution-spec-tests#1687

Control Flow

• [x] 0x56 JUMP ethereum/execution-spec-tests#1616
• [x] 0x57 JUMPI ethereum/execution-spec-tests#1690
• [x] 0x58 PC ethereum/execution-spec-tests#1651
• [x] 0x5B JUMPDEST ethereum/execution-spec-tests#1616

Stack Duplication

• [x] 0x80-0x8F DUP1-DUP16 (Does this make sense to add?) ethereum/execution-spec-tests#1687

Stack Swapping

• [x] 0x90-0x9F SWAP1-SWAP16 (Does this make sense to add?) ethereum/execution-spec-tests#1687

Logging

• [x] 0xA0 LOG0 ethereum/execution-spec-tests#1689
• [x] 0xA1 LOG1 ethereum/execution-spec-tests#1689
• [x] 0xA2 LOG2 ethereum/execution-spec-tests#1689
• [x] 0xA3 LOG3 ethereum/execution-spec-tests#1689
• [x] 0xA4 LOG4 ethereum/execution-spec-tests#1689

Contract Lifecycle

• [x] 0xF0 CREATE ethereum/execution-spec-tests#1689
• [x] 0xF3 RETURN ethereum/execution-spec-tests#1689
• [x] 0xF5 CREATE2 ethereum/execution-spec-tests#1689
• [x] 0xFD REVERT ethereum/execution-spec-tests#1689
• [x] 0xFF SELFDESTRUCT ethereum/execution-spec-tests#1657

---

Precompiled Contracts

• [x] 0x01 ECDSA signature recovery ethereum/execution-spec-tests#1524
• [x] 0x02 SHA-256 hash function ethereum/execution-spec-tests#1524
• [x] 0x03 RIPEMD-160 hash function ethereum/execution-spec-tests#1524
• [x] 0x04 Identity function ethereum/execution-spec-tests#1524
• [x] 0x05 Modular exponentiation (EIP-198) ethereum/execution-spec-tests#1523
• [x] 0x06 Elliptic curve addition alt_bn128 (EIP-196) ethereum/execution-spec-tests#1581
• [x] 0x07 Elliptic curve scalar multiplication alt_bn128 (EIP-196) ethereum/execution-spec-tests#1581
• [x] 0x08 Elliptic curve pairing check alt_bn128 (EIP-197) ethereum/execution-spec-tests#1581
• [x] 0x09 BLAKE2f compression function (EIP-152) ethereum/execution-spec-tests#1581
• [x] 0x0A KZG Point evaluation precompile (EIP-4844) ethereum/execution-spec-tests#1581
• [x] (0x0B–0x11) BLS12-381 Curve Operations Precompile (EIP-2537) ethereum/execution-spec-tests#1581 - [x] (0x0B–0x11)
• [X] 0x100 P256VERIFY https://github.com/ethereum/execution-spec-tests/pull/1744

---

Misc

• [x] Empty block (We want an empty block to have some baseline) ethereum/execution-spec-tests#1650

[kevaundray]

A suggestion by @barnabasbusa is to also tests how the proving time changes with the number of blobs.

ie

• empty block, 0 blobs
• empty block, 32 blobs
• empty block, 48 blobs

Current hypothesis is that it should not change much given the EL validation logic for blobs is this: https://github.com/ethereum/EIPs/blob/master/EIPS/eip-4844.md#execution-layer-validation

But its something to validate with benchmarks

[jochem-brouwer]

0x00 STOP (Rationale: Cannot fill a block with STOP opcodes) 0xFE INVALID (Rationale: Cannot fill a block with INVALID opcodes)

I disagree, we can target contracts with *CALL* to target a contract which either STOPs or INVALIDs (forwarding 0 gas). This can be directly included in those tests.

[kevaundray]

0x00 STOP (Rationale: Cannot fill a block with STOP opcodes) 0xFE INVALID (Rationale: Cannot fill a block with INVALID opcodes)

I disagree, we can target contracts with *CALL* to target a contract which either STOPs or INVALIDs (forwarding 0 gas). This can be directly included in those tests.

Yeah what I wrote was imprecise -- calling a contract with just STOP opcodes would return immediately after encountering the first STOP opcode, so it should be equivalent to having just one STOP opcode -- do you think this is still worth adding to the benchmarks?

[jsign]

@kevaundray, @jochem-brouwer, we have this STOP case already covered in my warm-CALL-like test. See here.

What I do for warm-CALL-like is exactly that. Call a contract that starts with STOP, since I want as much gas as possible to be used for CALL-like and not other stuff.

[jochem-brouwer]

Ah ok, if wanted you could also insert a variant of that test which forwards 0 gas to the next call frame and instead of STOP execute INVALID. This should not alter the gas costs at all (since frame has 0 gas costs, so spending all costs in the frame will still spend the same). Since the result of the *CALL* opcode is popped this also does not matter.

However, this would thus test two opcodes in the same test which might not be wanted. (However it would be interesting to see the actual cycle difference here, since the net gas costs and the amount of opcodes executed here is the same -> the only "net difference" is the item pushed to stack after the *CALL*)

@pytest.mark.parametrize(
    "opcode",
    [
        Op.BALANCE,
        Op.EXTCODESIZE,
        Op.EXTCODEHASH,
        Op.CALL,
        Op.CALLCODE,
        Op.DELEGATECALL,
        Op.STATICCALL,
    ],
)
[email protected](
+   "frame_terminator",
+   [
+      Op.STOP,
+      Op.INVALID,
+   ]
+)

opcode: Op,
+frame_terminator: Op,

-iter_block = Op.POP(opcode(address=Op.MLOAD(0)))
+iter_block = Op.POP(opcode(gas=0,address=Op.MLOAD(0)))

    if not absent_target:
-        code = Op.STOP + Op.JUMPDEST * 100
+       code = frame_terminator + Op.JUMPDEST * 100

[jsign]

@jochem-brouwer, thanks for the idea :) ~But I think we should probably do gas=0 if frame_terminator=Op.INVALID else Op.GAS, if we don't do that we would be missing actually "executing" the STOP which shouldn't have a problem. For the "free frame" we can consider INVALID covering it.~

I'll take a note to iterate on that existing test so we can say that INVALID is covered.

[jochem-brouwer]

Should have mentioned that, but STOP costs 0 gas thus will also execute if gas=0.

[jsign]

You're right!

[jochem-brouwer]

Can the JUMP test be checked? It likely jumps to a non-jumpdest opcode. See: https://github.com/ethereum/execution-spec-tests/pull/1693#discussion_r2123023914

[jsign]

Can the JUMP test be checked? It likely jumps to a non-jumpdest opcode. See: #1693 (comment)

Ah, that one was created by Cody, but let I'll check.

[LouisTsai-Csie]

Should we have one test that a block is filled with 7702 setCode transaction?

Update: I create one here: https://github.com/ethereum/execution-spec-tests/pull/1758

[LouisTsai-Csie]

New precompile (P256VERIFY) and opcode (CLZ) are introduced in Fusaka.

Benchmark for P256VERIFY is implemented here, and this PR is for CLZ opcode.

However, the existing test case in Osaka would not work due to the transaction gas limit gap in EIP-7825, this is documented in the issue as well.

[jsign]

New precompile (P256VERIFY) and opcode (CLZ) are introduced in Fusaka.

Benchmark for P256VERIFY is implemented here, and this PR is for CLZ opcode.

However, the existing test case in Osaka would not work due to the transaction gas limit gap in EIP-7825, this is documented in the issue as well.

Added both to the list with linked PRs.

[jsign]

The only remaining one now is CLZ, but since this isn't still in mainnet, and is also being tracked in this other issue I think we can remove it from this original issue tracker.

[jsign]

Closing this since all mainnet opcodes and precompiles are covered.

Congrats everybody that was involved in this first big iteration!