Unexpected `Panic` Error and Increased Gas Consumption in zkSync Era Transactions

[mshakeg]

🐛 Bug Report:

📝 Description

I'm encountering two main issues with transactions on the zkSync Era mainnet.

First, some transactions exit with a Panic error such as this transaction, even though they have been allocated sufficient gas for execution. This error occurs under conditions that do not traditionally indicate an out-of-gas situation, as the gas used is significantly below the gas limit specified. Regarding this issue more detail is provided under Additional Context. NOTE: the aforementioned transaction and the 2 transactions mentioned below perform essentially the same operations(with slightly different state that shouldn't significantly change the gas requirement).

Secondly, there's a noticeable increase in gas consumption for transactions that execute essentially the same logic(which can be verified by observing similar call traces), observed over a span of 4 weeks, without any apparent changes to the transaction logic or smart contract code or state changes to warrant such a drastic increase in gas cost.

• Comparative Analysis of Gas Consumption:
  • Transaction from 4 weeks ago: 0xe55eb635aef7e4b2ba4241fdb9ddea7ed341dd2f959d708f1c65e35a3dd4a169
  • More Recent Transaction: 0x2e6d0b8ff88c1e528644f90ef2bd63c13e7a02921264991bcecd87c4fddb18c0
  • Observation: The gas consumption for the recent transaction increased significantly to almost 7M gas from just over 4M gas for an earlier transaction performing similar logic. This change occurred without apparent alterations in the transaction logic or smart contract code that could justify such an increase.

🔄 Reproduction Steps

1. Execute a transaction similar to the provided example.
2. Use the debug_traceTransaction method to analyze the transaction.
3. Observe the Panic error in the debug trace and compare gas consumption with similar transactions over time.

🤔 Expected Behavior

Transactions with sufficient gas limits should not exit prematurely with a Panic error. Similarly, transactions executing the same logic and interacting with the same contracts should have consistent gas consumption, allowing for minor fluctuations.

😯 Current Behavior

• Transactions are exiting with a Panic error, despite having sufficient gas allocated.
• A significant, unexplained increase in gas consumption for similar transactions has been observed over a relatively short period.

🖥️ Environment

• zkSync Era Mainnet
• Transactions observed over the last 4 weeks.

📋 Additional Context

The issue was identified through the analysis of transaction outcomes and gas consumption patterns. The Panic error was specifically observed in a transaction debug trace, indicating an unexpected termination of the transaction execution.

📎 Log Output

{
    "jsonrpc": "2.0",
    "result": {
        // ...
        "calls": [
            {
                "type": "Call",
                "from": "0xf670f7fa7e55ed9816fbe2685aeb1b4e963f923d",
                "to": "0x291d9f9764c72c9ba6ff47b451a9f7885ebf9977",
                "gas": "0x106815",
                "gasUsed": "0x106815",
                "value": "0x0",
                "output": "0x",
                "input": "...",
                "error": "Panic",
                "revertReason": null,
                "calls": [
                    // ...
                 ],
                 // ...
            }
        ]
    }
}

I executed the following command to retrieve the debug trace for the transaction exhibiting the Panic error:

curl https://mainnet.era.zksync.io \
-X POST \
-H "Content-Type: application/json" \
--data '{"method":"debug_traceTransaction","params":["0xf58a6a2c2d24ec84d8db481524b7ba0b4c833812f1dc2a2f99c8d00d8efc7a3a", {"tracer": "callTracer"}], "id":1,"jsonrpc":"2.0"}'

This issue raises concerns about potential underlying problems with gas estimation or execution logic on the zkSync Era mainnet.

[mshakeg]

This tx is a more extreme example of the Panic issue, the tx failed despite having way more than the required gas.

curl https://mainnet.era.zksync.io \
-X POST \
-H "Content-Type: application/json" \
--data '{"method":"debug_traceTransaction","params":["0x423cb497f52f5fa40b850f3b83d0c04c184cd92b55a44d84c1bd22aca31096ea", {"tracer": "callTracer"}], "id":1,"jsonrpc":"2.0"}'

[cl3pl4t3]

you need to add an input

[mshakeg]

@cl3pl4t3 can you elaborate? what do you mean by add an input?

[EmilLuta]

cc: @vladbochok

[perekopskiy]

Hi @mshakeg

First, some transactions exit with a Panic error such as this transaction, even though they have been allocated sufficient gas for execution.

Why do you think that there was enough gas for execution? gas_used < gas_limit is not a strong evidence as some gas is refunded.

A significant, unexplained increase in gas consumption for similar transactions has been observed over a relatively short period.

Gas per pubdata may fluctuate a lot, so this can't hold. Please refer to https://docs.zksync.io/zk-stack/concepts/fee-mechanism.html for more details

[mshakeg]

Hi @perekopskiy

Why do you think that there was enough gas for execution? gas_used < gas_limit is not a strong evidence as some gas is refunded.

On all other EVM networks that we've deployed to if a tx executes with gas_used << gas_limit and the tx didn't revert due to a standard evm revert then the tx executed successfully but the gas limit specified for the transaction was >> the gas required for the tx(i.e. gas_used). Are you saying that on zkSync it's possible for gas_used << gas_limit and to not evm revert, but also not execute to completion successfully? This seems like very atypical behaviour.

[perekopskiy]

Yes, on zkSync refunds are much higher than on other EVM chains. So, if your transaction runs out of gas you will be refunded some of it (in some cases it can be very close to gas_limit, that's likely what you're seeing)

[maxfundingrate]

Hi @mshakeg

First, some transactions exit with a Panic error such as this transaction, even though they have been allocated sufficient gas for execution.

Why do you think that there was enough gas for execution? gas_used < gas_limit is not a strong evidence as some gas is refunded.

A significant, unexplained increase in gas consumption for similar transactions has been observed over a relatively short period.

Gas per pubdata may fluctuate a lot, so this can't hold. Please refer to https://docs.zksync.io/zk-stack/concepts/fee-mechanism.html for more details

@perekopskiy what's a good practice here for estimating gas then?

We have the exact same issue and we use zks_estimateFee with a buffer (approx 150%) on zksync sepolia but still get what we think are out of gas errors.

I saw one that zks_estimateFee RPC call returns a gas_per_pubdata field, is there a way to incorporate this into our gas estimations (e.g. by some heuristic)?

Here's our code in web3py for sending transactions, could you please advise on what else we can do?

            nonce = await self.w3.eth.get_transaction_count(
                self.address, block_identifier="latest"
            )
            unsigned_txn_estimate = contract_function.build_transaction(
                {
                    "nonce": nonce,
                    "from": self.address,
                    "maxFeePerGas": hex(100000000),
                }
            )
            estimated_fees = self.zk_w3.zksync.zks_estimate_fee(unsigned_txn_estimate)
            unsigned_txn = contract_function.build_transaction(
                {
                    "nonce": nonce,
                    "from": self.address,
                    "gas": estimated_fees.gas_limit,
                    "maxFeePerGas": estimated_fees.max_fee_per_gas,
                    "maxPriorityFeePerGas": estimated_fees.max_priority_fee_per_gas,
                }
            )
            signed_txn = self.w3.eth.account.sign_transaction(
                unsigned_txn, private_key=self.private_key
            )
            txn_hash = await self.w3.eth.send_raw_transaction(signed_txn.rawTransaction)

[mshakeg]

@perekopskiy I see, what is the max gas refund percentage, on Ethereum mainnet it's 50%.

[perekopskiy]

@mshakeg max gas refund percentage is 100%. Gas estimation endpoint returns gas enough for high-level transaction to succeed. It doesn't take into account that some low-lowel call may fail with out of gas. It works same way on Ethereum. All transactions you shared were successful

[mshakeg]

@perekopskiy does gas estimation work internally? i.e. if I'm estimating gas for calls using a static eth_call to multicall, will that return the same gas estimation for eth_estimateGas, or does eth_estimateGas do some additional computation on the evm level gas estimation resulting in a different value?

[perekopskiy]

@mshakeg sorry, I don't understand the question

[mshakeg]

@perekopskiy so to estimate gas instead of using eth_estimateGas I'm using an static call(i.e. eth_call) to the multicall, and for each Call I am specifying a high gasLimit say 10m as I'm confident that the Call will use somewhere between 1m and 4m, now Result for a given Call will return gasUsed which is computed as the difference between gasleft() after and gasleft() before a given Call was executed. Now is this gasUsed identical to the gas estimate returned by an eth_estimateGas request for that same Call or would it be different as maybe eth_estimateGas returns some other value that is a function of gasUsed and possibly other parameters specific to zksync?

[perekopskiy]

I see, they won't be same but should be pretty close, estimate gas will add intristic tx gas, tx overhead, etc. Note, there are cases where they will differ much, so it's recommended to use eth_estimateGas.

[mshakeg]

@perekopskiy hmm, could you please elaborate on the cases where they will differ much?

[perekopskiy]

@mshakeg

• if you use custom account and validation takes a lot of gas
• if refund is high: it can be if you write some values to storage slots and then rollback them, e.g. if you use reentrancy lock

[EmilLuta]

@mshakeg can we close this ticket, or do you need further assistance?

[mshakeg]

@EmilLuta haven't had to test if the issue still persists recently, might have to in future. But if no changes have been made to address this issue I suspect it's still an issue.

[EBRAHIM1997Z]

👍👍