Enhance Support for Hardhat Network Forking

[mshakeg]

Problem

Developers utilizing Hardhat(or Foundry) for Ethereum development often rely on the forking feature to test their contracts and applications against live networks. Given that HIP-584 is final there should be no major blockers to network forking. However, when attempting to use this feature against the Hedera testnet relay, I've encountered compatibility issues. Running a fork test that deploys a contract and interacts with that contract and other contracts already live on testnet results in at least the following error:

Errors: Invalid value "0x095" supplied to : RpcBlockWithTransactions | null/transactions: RpcTransaction Array/0: RpcTransaction/gasPrice: QUANTITY

Solution

The Hedera team should conduct a thorough investigation to identify and resolve any incompatibilities or bugs that cause Hardhat fork tests to fail. This investigation should aim to ensure full support for Hardhat&Foundry network forking.

Alternatives

No response

[Ivo-Yankov]

Hi @mshakeg ,

For fork tests require batch requests to be enabled on the relay. Unfortunatelly that is not the case with the public relays, but you can run a local instance and configure it to work with testnet or mainnet. This env variable enables batch requests:

BATCH_REQUESTS_ENABLED=true

I did some research and found that fork tests work for hardhat up to version 2.20.1, but not for the latest versions. With Foundry it seems to be working fine. You can see an example in this repo

[mshakeg]

Hi @Ivo-Yankov why are batch requests not enabled on the hashio relay? It makes for a terrible dev experience, all the other EVM chains that I've interacted with have public relays that fully support&enable all the functionality required for fork tests.

[AlfredoG87]

Hello @Ivo-Yankov and @mshakeg ,

I can confirm that BATCH_REQUESTS_ENABLED is enabled on all production environments of HashIO. (and we will enable it by default on local too, soon)

That said, I don't think we support Forking of the blockchain at a given snapshot just yet. we are working on that feature and will update once we confirm that we have the capabilities.

Thanks

[mshakeg]

Hi @AlfredoG87 thanks for clarifying. Yes it doesn't seem like forking at a given snapshot(block height) is supported at least not fully as my fork tests fail with errors such as the one in the issue description.

[bguiz]

This example is highly-specific to hardhat, but I think might shed some light on how this could be implemented in Hedera: https://stackoverflow.com/a/78191980/194982

Notably:

• The file hardhat/internal/hardhat-network/jsonrpc/client.js in this example performs a similar function to Hedera RPC Relay, in the sense that it proxies/ translates RPC requests.
• In the case of hardhat, it's between the client and any RPC endpoint configured (including a faux EVM implementation, in memory). In the case of Hedera RPC Relay, it is between the client and a combination of Hedera Mirror Nodes and Hedera Consensus Nodes.
• In that context, we could re-use some ideas from this patch to hardhat within Hedera RPC Relay.

Specifically:

• See the _patchRawResult function that has been implemented.
  • It identifies specific differences between Ethereum's EVM implementations (e.g. geth) RPC responses, and the version the the rootstock node
  • Subsequently it applies some simple substitution and formatting differences to the RPC response before relaying the response to the client
• In Hedera RPC Relay, we could implement an equivalent of the _patchRawResult function.
  • Within that, we could implement some formatting/ substitution for the already known/ identified differences (e.g. stateRoot)
  • We could also apply fuzz testing (or other suitable untrained discovery methodologies) to identify differences between responses between Hedera's RPC Relay and RPC endpoints from other nodes; and subsequently implement some formatting/ substitution for these too.

Refs:

• Prior discussions:

  • hardcoded zero value for stateRoot in RPC responses for eth_getBlockBy* RPC requests https://swirldslabs.slack.com/archives/C03EHC2H8CR/p1706056921483199

  • https://github.com/hashgraph/hedera-docs/issues/562
• On fuzz testing:
  • https://github.com/hashgraph/hedera-json-rpc-relay/issues/2066
  • https://blocksec.com/blog/security-check-do-evm-compatible-chains-hold-up
  • https://blocksecteam.medium.com/systematic-approach-to-maintaining-evm-compatibility-and-security-32d0f42846f5
  • Patch for hardhat to enable Hardhat network forking in Rootstock: https://stackoverflow.com/a/78191980/194982

cc @Nana-EC @Neurone @a-ridley @theekrystallee

[Ivo-Yankov]

Hi @mshakeg, can you please confirm if your tests are working with [email protected] with and without specifying the block number?

[mshakeg]

Hi @Ivo-Yankov it almost works entirely, however it fails when the script attempts to estimate gas(eth_estimateGas) on the deployment of a contract like this one. The contract is verified so you should be able to replicate the issue.

[mshakeg]

Hi @AlfredoG87 it seems like BATCH_REQUESTS_ENABLED has been set to false on the hashio mainnet relay? I'm getting the following response now:

Errors: Invalid value null supplied to : RpcBlockWithTransactions | null/transactions: RpcTransaction Array/0: RpcTransaction/gasPrice: QUANTITY, Invalid value null supplied to : RpcBlockWithTransactions | null/transactions: RpcTransaction Array/0: RpcTransaction/v: QUANTITY, Invalid value null supplied to : RpcBlockWithTransactions | null/transactions: RpcTransaction Array/0: RpcTransaction/r: QUANTITY, Invalid value null supplied to : RpcBlockWithTransactions | null/transactions: RpcTransaction Array/0: RpcTransaction/s: QUANTITY, Invalid value null supplied to : RpcBlockWithTransactions | null/transactions: RpcTransaction Array/0: RpcTransaction/type: QUANTITY | undefined

[AlfredoG87]

Hi @AlfredoG87 it seems like BATCH_REQUESTS_ENABLED has been set to false on the hashio mainnet relay? I'm getting the following response now:

@mshakeg I've just verified and BATCH_REQUESTS_ENABLED is seems to be enabled on mainnet

Could you share your request so I can reproduce it on my end?

I just made a batch request using this request and the response is as expected and working:

Request:

curl --location 'https://mainnet.hashio.io/api' \
--header 'Content-Type: application/json' \
--data '[
    {
        "jsonrpc": "2.0",
        "id": 1,
        "method": "eth_chainId",
        "params": []
    },
    {
        "jsonrpc": "2.0",
        "id": 2,
        "method": "eth_chainId",
        "params": []
    },
    {
        "id": 2,
        "jsonrpc": "2.0",
        "method": "eth_blockNumber"        
    }
]'

Response:

[
    {
        "result": "0x127",
        "jsonrpc": "2.0",
        "id": 1
    },
    {
        "result": "0x127",
        "jsonrpc": "2.0",
        "id": 2
    },
    {
        "result": "0x3bb1f1c",
        "jsonrpc": "2.0",
        "id": 2
    }
]```

[mshakeg]

@AlfredoG87 it fails on the following line in a hardhat script. Not sure what all the json rpc requests involved in this are, but it worked fine in a fork test about 2 weeks ago.

import { ethers } from "hardhat";

// other code ...

const signers = await ethers.getSigners(); // fails on this line

[AlfredoG87]

@AlfredoG87 it fails on the following line in a hardhat script. Not sure what all the json rpc requests involved in this are, but it worked fine in a fork test about 2 weeks ago.

import { ethers } from "hardhat";

// other code ...

const signers = await ethers.getSigners(); // fails on this line

I am not sure what is the cause of the error but seems to me unrelated to BATCH_REQUESTS_ENABLED or even the relay. seems to me like a configuration issue is the culprit on your hardhat project.

If you could provide a barebone minimal project where the issue is happening, I would be happy to take a look.

[mshakeg]

@AlfredoG87 sure, below is a minimal reproduction of the issue, it seems like it could be a pruning issue, as forking from a block a few days ago fails but from a recent block today doesn't, so maybe the mirror node needs to keep a lot more history(at least 1 month ago)?

https://github.com/mshakeg/hardhat-template/pull/2

[mshakeg]

Hi @AlfredoG87 I just wanted to follow up on my last reply. Have you made any progress?

[mshakeg]

Hi @Ivo-Yankov just wanted to follow up on this reply. Do you know if the public mirror node prunes contract state and if so how far back does it go?