Integration test failed

[wangtsiao]

What command(s) is the bug in?

cargo test --release --test circuits real_prover::serial_test_super_circuit_multiple_erc20_openzeppelin_transfers -- --nocapture 2>&1

Describe the bug

mock_prover passes tests, but the following error occurs when switching to real_prover.

Not only circuit_multiple_erc20_openzeppelin_transfers, but also circuit_block_transfer_0, circuit_multiple_transfers_0, and circuit_erc20_openzeppelin_transfer_succeed.

cargo test --release --test circuits real_prover::serial_test_super_circuit_multiple_erc20_openzeppelin_transfers -- --nocapture 2>&1
    Finished release [optimized] target(s) in 0.77s
     Running tests/circuits.rs (/home/root/zkevm-circuits/target/release/deps/circuits-7a8fd0c23e232ec8)

running 1 test
[2023-05-30T07:34:55Z INFO  integration_tests::integration_test_circuits] test Super circuit, block: #8 - Multiple ERC20 OpenZeppelin transfers
test real_prover::serial_test_super_circuit_multiple_erc20_openzeppelin_transfers has been running for over 60 seconds
thread 'real_prover::serial_test_super_circuit_multiple_erc20_openzeppelin_transfers' panicked at 'proof generation should not fail: ConstraintSystemFailure', /home/root/zkevm-circuits/integration-tests/src/integration_test_circuits.rs:192:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test real_prover::serial_test_super_circuit_multiple_erc20_openzeppelin_transfers ... FAILED

failures:

failures:
    real_prover::serial_test_super_circuit_multiple_erc20_openzeppelin_transfers

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 79 filtered out; finished in 1182.33s

error: test failed, to rerun pass `--test circuits`

Concrete steps to reproduce the bug. If it's able reproduce via testool, please share test_id from jenkins report

cd integration-tests
cargo test --release --test circuits real_prover::serial_test_super_circuit_multiple_erc20_openzeppelin_transfers -- --nocapture 2>&1

[ed255]

Thanks for the report, we're currently investigating this.

I have run the integration tests for each individual circuit with the real prover and they all seem to pass.

I used this script:

CIRCUITS="bytecode_circuit
copy_circuit
evm_circuit
exp_circuit
keccak_circuit
state_circuit
tx_circuit"
# super_circuit"

TESTS="block_transfer_0
multiple_transfers_0
erc20_openzeppelin_transfer_succeed
erc20_openzeppelin_transfer_fail
multiple_erc20_openzeppelin_transfers"

for circuit in $CIRCUITS; do
    for test in $TESTS; do
        cargo test --release --test circuits real_prover::serial_test_${circuit}_${test} -- --nocapture
    done
done

This are the results:

Click to expand

running 1 test
test real_prover::serial_test_bytecode_circuit_block_transfer_0 ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 9.45s


running 1 test
test real_prover::serial_test_bytecode_circuit_multiple_transfers_0 ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 9.30s


running 1 test
test real_prover::serial_test_bytecode_circuit_erc20_openzeppelin_transfer_succeed ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 9.64s


running 1 test
test real_prover::serial_test_bytecode_circuit_erc20_openzeppelin_transfer_fail ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 9.64s


running 1 test
test real_prover::serial_test_bytecode_circuit_multiple_erc20_openzeppelin_transfers ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 9.62s


running 1 test
test real_prover::serial_test_copy_circuit_block_transfer_0 ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 15.26s


running 1 test
test real_prover::serial_test_copy_circuit_multiple_transfers_0 ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 15.16s


running 1 test
test real_prover::serial_test_copy_circuit_erc20_openzeppelin_transfer_succeed ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 15.51s


running 1 test
test real_prover::serial_test_copy_circuit_erc20_openzeppelin_transfer_fail ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 15.44s


running 1 test
test real_prover::serial_test_copy_circuit_multiple_erc20_openzeppelin_transfers ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 15.57s


running 1 test
test real_prover::serial_test_evm_circuit_block_transfer_0 has been running for over 60 seconds
test real_prover::serial_test_evm_circuit_block_transfer_0 ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 461.67s


running 1 test
test real_prover::serial_test_evm_circuit_multiple_transfers_0 has been running for over 60 seconds
test real_prover::serial_test_evm_circuit_multiple_transfers_0 ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 462.29s


running 1 test
test real_prover::serial_test_evm_circuit_erc20_openzeppelin_transfer_succeed has been running for over 60 seconds
test real_prover::serial_test_evm_circuit_erc20_openzeppelin_transfer_succeed ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 462.89s


running 1 test
test real_prover::serial_test_evm_circuit_erc20_openzeppelin_transfer_fail has been running for over 60 seconds
test real_prover::serial_test_evm_circuit_erc20_openzeppelin_transfer_fail ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 466.73s


running 1 test
test real_prover::serial_test_evm_circuit_multiple_erc20_openzeppelin_transfers has been running for over 60 seconds
test real_prover::serial_test_evm_circuit_multiple_erc20_openzeppelin_transfers ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 463.89s


running 1 test
test real_prover::serial_test_exp_circuit_block_transfer_0 ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 5.38s


running 1 test
test real_prover::serial_test_exp_circuit_multiple_transfers_0 ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 5.39s


running 1 test
test real_prover::serial_test_exp_circuit_erc20_openzeppelin_transfer_succeed ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 5.38s


running 1 test
test real_prover::serial_test_exp_circuit_erc20_openzeppelin_transfer_fail ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 5.34s


running 1 test
test real_prover::serial_test_exp_circuit_multiple_erc20_openzeppelin_transfers ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 5.42s


running 1 test
test real_prover::serial_test_keccak_circuit_block_transfer_0 has been running for over 60 seconds
test real_prover::serial_test_keccak_circuit_block_transfer_0 ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 76.31s


running 1 test
test real_prover::serial_test_keccak_circuit_multiple_transfers_0 has been running for over 60 seconds
test real_prover::serial_test_keccak_circuit_multiple_transfers_0 ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 77.55s


running 1 test
test real_prover::serial_test_keccak_circuit_erc20_openzeppelin_transfer_succeed has been running for over 60 seconds
test real_prover::serial_test_keccak_circuit_erc20_openzeppelin_transfer_succeed ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 95.78s


running 1 test
test real_prover::serial_test_keccak_circuit_erc20_openzeppelin_transfer_fail has been running for over 60 seconds
test real_prover::serial_test_keccak_circuit_erc20_openzeppelin_transfer_fail ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 92.32s


running 1 test
test real_prover::serial_test_keccak_circuit_multiple_erc20_openzeppelin_transfers has been running for over 60 seconds
test real_prover::serial_test_keccak_circuit_multiple_erc20_openzeppelin_transfers ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 97.02s


running 1 test
test real_prover::serial_test_state_circuit_block_transfer_0 has been running for over 60 seconds
test real_prover::serial_test_state_circuit_block_transfer_0 ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 73.43s


running 1 test
test real_prover::serial_test_state_circuit_multiple_transfers_0 has been running for over 60 seconds
test real_prover::serial_test_state_circuit_multiple_transfers_0 ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 73.68s


running 1 test
test real_prover::serial_test_state_circuit_erc20_openzeppelin_transfer_succeed has been running for over 60 seconds
test real_prover::serial_test_state_circuit_erc20_openzeppelin_transfer_succeed ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 79.10s


running 1 test
test real_prover::serial_test_state_circuit_erc20_openzeppelin_transfer_fail has been running for over 60 seconds
test real_prover::serial_test_state_circuit_erc20_openzeppelin_transfer_fail ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 79.03s


running 1 test
test real_prover::serial_test_state_circuit_multiple_erc20_openzeppelin_transfers has been running for over 60 seconds
test real_prover::serial_test_state_circuit_multiple_erc20_openzeppelin_transfers ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 79.19s


running 1 test
test real_prover::serial_test_tx_circuit_block_transfer_0 has been running for over 60 seconds
test real_prover::serial_test_tx_circuit_block_transfer_0 ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 221.28s


running 1 test
test real_prover::serial_test_tx_circuit_multiple_transfers_0 has been running for over 60 seconds
test real_prover::serial_test_tx_circuit_multiple_transfers_0 ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 222.70s


running 1 test
test real_prover::serial_test_tx_circuit_erc20_openzeppelin_transfer_succeed has been running for over 60 seconds
test real_prover::serial_test_tx_circuit_erc20_openzeppelin_transfer_succeed ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 220.84s


running 1 test
test real_prover::serial_test_tx_circuit_erc20_openzeppelin_transfer_fail has been running for over 60 seconds
test real_prover::serial_test_tx_circuit_erc20_openzeppelin_transfer_fail ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 221.93s


running 1 test
test real_prover::serial_test_tx_circuit_multiple_erc20_openzeppelin_transfers has been running for over 60 seconds
test real_prover::serial_test_tx_circuit_multiple_erc20_openzeppelin_transfers ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 221.62s

[ed255]

Idea: we can compare the status of this issue with the scroll fork (to see if we're failing the same tests or not)

[AronisAt79]

Thanks @ed255 for the analysis. integration tests in e2e environment are for the most part successful, apart from the EVM and SUPER circuit proofs, Reference issues : #1334 #1335 (all tests run with aggregate enabled) Super circuit test without aggregate is currently in progress. Will update here as soon as test concludes.

[ed255]

Here's a summary of what we have tested so far:

Results for real prover with:

• integration tests (standalone) commit 73b2aebca9400ac8b31c9394fb726690dbfe0b74
• zkevm-chain tests (with Aggregation) branch main at 2023-05-30 02:00:00

Circuit	standalone	with Aggregation
EVM	PASS	FAILED
State	PASS	PASS
Tx	PASS	PASS
Bytecode	PASS	PASS
Copy	PASS	PASS
Keccak	PASS	PASS
Exp	PASS	PASS
Pi	PASS	PASS
Super	FAILED	FAILED

[AronisAt79]

proof for super circuit / no aggregate succeeds. https://zkevm-chain-testing.s3.eu-central-1.amazonaws.com/proveSingeCrossChainTx_super_False_410-maronis.tar.gz

[ed255]

We have been able to reproduce this issue. All integration tests with the real prover and super circuit are failing https://github.com/privacy-scaling-explorations/zkevm-circuits/actions/runs/5152993138/jobs/9279691434

The fact that the subcircuits test with real prover pass, and that all mock prover tests pass makes it a bit hard to debug this issue. A ConstraintSystemFailure is very opaque and doesn't give you any hint of what may be the problem (which ideally should be found by the MockProver, but that's not happening)

[ed255]

proof for super circuit / no aggregate succeeds. https://zkevm-chain-testing.s3.eu-central-1.amazonaws.com/proveSingeCrossChainTx_super_False_410-maronis.tar.gz

Update on this: The test that succeeded was using an old version of the zkevm-circuits. Via the integration test run via github actions we have been able to reproduce the issue and observe the failure (see previous message)