py-evm
py-evm copied to clipboard
Weird balance overrides when forking `py-evm`
- py-evm Version: 0.3.0a19
- OS: osx
- Python Version (python --version): 3.7
- Environment (output of
pip freeze
):
Pip freeze output
aniso8601==7.0.0
appdirs==1.4.4
appnope==0.1.0
argon2-cffi==20.1.0
asttokens==2.0.3
async-generator==1.10
attrdict==2.0.1
attrs==20.2.0
Babel==2.8.0
backcall==0.2.0
base58==2.0.1
bitarray==1.2.2
black==20.8b1
blake2b-py==0.1.3
bleach==3.2.1
cached-property==1.5.2
certifi==2020.6.20
cffi==1.14.3
chardet==3.0.4
click==7.1.2
cytoolz==0.11.0
decorator==4.4.2
defusedxml==0.6.0
entrypoints==0.3
eth-abi==2.1.1
eth-account==0.5.4
eth-bloom==1.0.3
eth-hash==0.2.0
eth-keyfile==0.5.1
eth-keys==0.3.3
eth-rlp==0.2.1
eth-tester==0.5.0b2
eth-typing==2.2.2
eth-utils==1.9.5
fastdiff==0.2.0
Flask==1.1.2
Flask-Cors==3.0.9
Flask-GraphQL==2.0.1
future==0.18.2
gevent==20.9.0
graphene==2.1.8
graphql-core==2.3.2
graphql-relay==2.0.1
graphql-server-core==1.2.0
greenlet==0.4.17
gunicorn==20.0.4
hexbytes==0.2.1
idna==2.10
importlib-metadata==2.0.0
iniconfig==1.0.1
ipfshttpclient==0.6.1
ipykernel==5.3.4
ipython==7.18.1
ipython-genutils==0.2.0
isort==5.5.3
itsdangerous==1.1.0
jedi==0.17.2
Jinja2==2.11.2
json5==0.9.5
jsonschema==3.2.0
jupyter-client==6.1.7
jupyter-core==4.6.3
jupyter-server==1.0.0rc16
jupyterlab==2.2.8
jupyterlab-pygments==0.1.1
jupyterlab-server==1.2.0
lru-dict==1.1.6
MarkupSafe==1.1.1
mistune==0.8.4
more-itertools==8.5.0
multiaddr==0.0.9
mypy==0.782
mypy-extensions==0.4.3
nbclassic==0.2.0rc7
nbclient==0.5.0
nbconvert==6.0.5
nbformat==5.0.7
nest-asyncio==1.4.0
netaddr==0.8.0
notebook==6.1.4
packaging==20.4
pandocfilters==1.4.2
parsimonious==0.8.1
parso==0.7.1
pathspec==0.8.0
pexpect==4.8.0
pickleshare==0.7.5
pluggy==0.13.1
prometheus-client==0.8.0
promise==2.3
prompt-toolkit==3.0.7
protobuf==3.13.0
ptyprocess==0.6.0
py==1.9.0
py-ecc==4.1.0
py-evm==0.3.0a19
py-geth==2.4.0
py-solc===3.2.0-fixedstdin
py-solc-x==1.0.0
pycparser==2.20
pycryptodome==3.9.8
pyethash==0.1.27
pyevmasm==0.2.3
Pygments==2.7.1
pyparsing==2.4.7
pyrsistent==0.17.3
pysha3==1.0.2
pytest==6.0.2
pytest-tornasync==0.6.0.post2
python-dateutil==2.8.1
python-dotenv==0.14.0
python-json-logger==2.0.1
pytz==2020.1
pyzmq==19.0.2
regex==2020.7.14
requests==2.24.0
rlp==2.0.0a1
rusty-rlp==0.1.15
Rx==1.6.1
semantic-version==2.8.5
Send2Trash==1.5.0
six==1.15.0
snapshottest==0.5.1
sortedcontainers==2.2.2
termcolor==1.1.0
terminado==0.9.1
testpath==0.4.4
toml==0.10.1
toolz==0.11.1
tornado==6.0.4
traitlets==5.0.4
trie==2.0.0a4
typed-ast==1.4.1
typing-extensions==3.7.4.3
urllib3==1.25.11
varint==1.0.2
vyper==0.2.7
wasmer==0.4.1
wcwidth==0.2.5
web3==5.12.2
webencodings==0.5.1
websockets==8.1
Werkzeug==1.0.1
zipp==3.3.1
zope.event==4.5.0
zope.interface==5.1.2
What is wrong?
I'm running a notebook shown below (exported in md
for simplicity).
More about how this works:
- This is a
py-evm
fork that attempts to override the balance for a single address (the coinbase account) - Initially the address seems to be overridden, but when we call
getBalance
on the account, it reverts back to 0, any thoughts on why this may be the case? Note the lastget_balance
callupdated get_balance(0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf) = 0
which comes after setting the balance for that addressset_balance(0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf) := 2000000000000000000
.
How is the fork itself set up:
- I'm forking the
get_balance
andset_balance
calls on the_account_db
- set balance is very simple, it just marks which addresses have been modified in
balances_set
, in addition, it ignores the first run (which is the initial balance set for the coinbase account) - get balance works normally, but if the coinbase address has not been set yet, it returns a preset value (2 * 10^18).
Fork testing notebook
Designed to explore why account balances do not preserve correctly when forking py-evm
.
from typing import Any, Iterable, Optional, Type
from eth.abc import BlockAPI, ExecutionContextAPI
from eth.rlp.blocks import BaseBlock
from eth.vm.forks.muir_glacier import MuirGlacierVM
from eth.vm.forks.muir_glacier.blocks import MuirGlacierBlock
from eth.vm.forks.muir_glacier.headers import (
compute_muir_glacier_difficulty,
configure_muir_glacier_header,
create_muir_glacier_header_from_parent,
)
from eth.vm.forks.muir_glacier.state import MuirGlacierState
from eth.vm.state import BaseState
from eth_hash.auto import keccak
from eth_typing import Address, BlockNumber, Hash32
from eth_utils import to_checksum_address
def get_fallback_vm_configuration():
balances_set = set()
ran = [0]
class FallbackState(MuirGlacierState):
def get_balance(self, address: Address) -> int:
adx = to_checksum_address(address.hex())
if address in balances_set or adx != "0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf":
balance = self._account_db.get_balance(address)
if address in balances_set:
print(f"updated get_balance({adx}) = {balance}")
else:
print(f"original get_balance({adx}) = {balance}")
return balance
# New idea: commit fallback balance to current chain when retrieving
if adx != "0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf":
raise Exception("impossible")
balance = 2 * 10**18
self.set_balance(address, balance)
print(f"override get_balance({adx}) = {balance}")
return balance
def set_balance(self, address: Address, balance: int) -> None:
adx = to_checksum_address(address.hex())
if ran[0] < 1 and adx == "0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf":
ran[0] += 1
print(f"ignored set_balance({adx}) := {balance}")
return # ignore first run
print(f"set_balance({adx}) := {balance}")
balances_set.add(address)
self._account_db.set_balance(address, balance)
class FallbackVM(MuirGlacierVM):
"""Fallback virtual machine that forks from a given web3 network."""
# fork name
fork = "muir-glacier-mainnet-fallback"
# classes
block_class: Type[BaseBlock] = MuirGlacierBlock
_state_class: Type[BaseState] = FallbackState
# Methods
create_header_from_parent = staticmethod(create_muir_glacier_header_from_parent) # type: ignore
compute_difficulty = staticmethod(compute_muir_glacier_difficulty) # type: ignore
configure_header = configure_muir_glacier_header
no_proof_vms = ((0, FallbackVM),)
return no_proof_vms
import web3
import eth_tester
vm_configuration = get_fallback_vm_configuration()
backend = eth_tester.PyEVMBackend(vm_configuration=vm_configuration)
w3 = web3.Web3(web3.Web3.EthereumTesterProvider(backend))
ignored set_balance(0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf) := 1000000000000000000000000
set_balance(0x2B5AD5c4795c026514f8317c7a215E218DcCD6cF) := 1000000000000000000000000
set_balance(0x6813Eb9362372EEF6200f3b1dbC3f819671cBA69) := 1000000000000000000000000
set_balance(0x1efF47bc3a10a45D4B230B5d10E37751FE6AA718) := 1000000000000000000000000
set_balance(0xe1AB8145F7E55DC933d51a18c793F901A3A0b276) := 1000000000000000000000000
set_balance(0xE57bFE9F44b819898F47BF37E5AF72a0783e1141) := 1000000000000000000000000
set_balance(0xd41c057fd1c78805AAC12B0A94a405c0461A6FBb) := 1000000000000000000000000
set_balance(0xF1F6619B38A98d6De0800F1DefC0a6399eB6d30C) := 1000000000000000000000000
set_balance(0xF7Edc8FA1eCc32967F827C9043FcAe6ba73afA5c) := 1000000000000000000000000
set_balance(0x4CCeBa2d7D2B4fdcE4304d3e09a1fea9fbEb1528) := 1000000000000000000000000
w3.eth.coinbase
'0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf'
w3.eth.getBalance(w3.eth.coinbase)
set_balance(0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf) := 2000000000000000000
override get_balance(0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf) = 2000000000000000000
2000000000000000000
w3.eth.accounts
['0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf',
'0x2B5AD5c4795c026514f8317c7a215E218DcCD6cF',
'0x6813Eb9362372EEF6200f3b1dbC3f819671cBA69',
'0x1efF47bc3a10a45D4B230B5d10E37751FE6AA718',
'0xe1AB8145F7E55DC933d51a18c793F901A3A0b276',
'0xE57bFE9F44b819898F47BF37E5AF72a0783e1141',
'0xd41c057fd1c78805AAC12B0A94a405c0461A6FBb',
'0xF1F6619B38A98d6De0800F1DefC0a6399eB6d30C',
'0xF7Edc8FA1eCc32967F827C9043FcAe6ba73afA5c',
'0x4CCeBa2d7D2B4fdcE4304d3e09a1fea9fbEb1528']
starting_bal = w3.eth.getBalance(w3.eth.accounts[0])
updated get_balance(0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf) = 0
How can it be fixed
This may not be a bug, but I'd love to hear how to ensure that the balances stay consistent between transactions if I do override them.
I haven't dived deep into what's going on with this code yet. This isn't really an answer to your question, but an alternative would be to follow the model of TheDAO fork: https://github.com/ethereum/py-evm/blob/239c72c202cb6371fd0059436152fda91592124e/eth/vm/forks/homestead/headers.py#L85-L105
@carver this makes sense and thanks so much for highlighting the example, it's quite elegant. Unfortunately, I'm making a dynamic fork to simulate mainnet calls so I'm following an approach similar to ganache
which responds and fills in gaps lazily as they are requested by specific transactions rather than pre-loading state at once. The idea is that without running the transactions in the first place, I don't really know what state to set.
Did you ever figure this out?
Unfortunately I didn't, I parked this until I have more observability infrastructure to be able to debug it better but one thing that helped with related issues was being thorough in incorporating all the API functions. For example for nonces, in addition to set_nonce
and get_nonce
there is increment_nonce
and important to update all of them.
@flux627 are you running into a similar issue?
I'm still in a research phase for tooling- looking to see if it was possible to fork mainnet with this like Ganache, in hopes that this implementation is faster. But, it seems that this tooling isn't really meant for this. Also looking at hevm, but I don't know Haskell and it doesn't have any bindings. Any suggestions for performant mainnet forking tests are welcome.
Unfortunately, we aren't currently putting any resources toward new features like this in py-evm (though forking mainnet is definitely a cool one that we've talked about, and would like some day).
Though it's fairly straightforward to think of "forking mainnet" (without having the full state) as a kind of variant of Beam Sync. So you can check out how Beam Sync is implemented in trinity, especially the pausing_vm_decorator
and how it overwrites VMState
. Note that it overrides all the methods (like increment_nonce
) in a similar way.
See https://github.com/ethereum/trinity/blob/eaa3b040ffdf0848b8e00a5f329480662ecc7c11/trinity/sync/beam/importer.py#L190