crytic
User features comparison with echidna / building secure contracts
Let's make a table comparing Echidna with Medusa, from a user standpoint. For example what config option/flags are identical, which ones are not (ex: callSequenceLength versus seqLen).
While doing this, we should look at https://github.com/crytic/building-secure-contracts/tree/master/program-analysis/echidna, and evaluate what exercises:
- work out of the box
- need a tweak on the config option
- do not work yet
Once we have this, we can prepare an update on building-secure-contracts to include Medusa
In the meantime, we should also keep a private copy of the echidna tests in building secure contracts, to make sure they pass during the development.
| Exercise | Status | Notes |
|---|---|---|
| 1 | need a tweak on the config option (testPrefixes, deploymentOrder) |
Add the echidna_ prefix (or use medusa's default fuzz_). Also, there is a need to set up the deploymentOrder option (to TestToken). |
| 2 | need a tweak on the config option (testPrefixes, deploymentOrder) |
Add the echidna_ prefix (or use medusa's default fuzz_). Also, there is a need to set up the deploymentOrder option (to TestToken). |
| 3 | need a tweak on the config option (testPrefixes, deploymentOrder) |
Add the echidna_ prefix (or use medusa's default fuzz_). Also, there is a need to set up the deploymentOrder option (to TestToken). |
| 4 | need a tweak on the config option (deploymentOrder, assertionTesting) |
Enable the assertionTesting and set up the deploymentOrder to Token |
| 5 | does not work yet | no config option for setting ETH in a contract |
| 6 | need a tweak on the config option (testPrefixes, deploymentOrder) |
Add the echidna_ prefix (or use medusa's default fuzz_). Also, there is a need to set up the deploymentOrder option (e.g.,UnstoppableEchidna). |
| 7 | does not work yet | no config option for setting ETH in a contract |
| 8 | does not work yet | ERROR: contract deployment tx returned a failed status: execution reverted, probably because of the unsupported max code size (echidna's codeSize) option |
Fuzzing Configuration
| Medusa | Echidna | Notes |
|---|---|---|
| workers | n/a | |
| workerResetLimit | n/a | |
| - | multi-abi | not implemented in Medusa |
| - | codeSize | not implemented in Medusa |
| timeout | - | not found in Echidna |
| testLimit | testLimit | |
| callSequenceLength | seqLen | |
| coverageEnabled | coverage | |
| corpusDirectory | corpusDir | |
| deploymentOrder | - | unsupported in Echidna (but you can always create Solidity code that deploys in the order you want. You can also easily specify constructor arguments there (or even, reuse some address, which is very common - G.G) |
| deployerAddress | deployer | |
| senderAddresses | sender, psender | psender (address of the sender of the property to test) is in the TODO in Medusa |
| blockNumberDelayMax | maxBlockDelay | |
| blockTimestampDelayMax | maxTimeDelay | |
| blockGasLimit | (?) | |
| transactionGasLimit | maxGasprice | |
| stopOnFailedTest | stopOnFail | |
| assertionTesting (enabled (bool), testViewMethods (bool)) | testMode: "assertion" | Medusa has a more detailed configuration - testViewMethods describes whether pure / view functions should be tested for assertion failures. |
| propertyTesting (enabled (bool), testPrefixes (String]) | testMode: "property" | Medusa has a more detailed configuration - testPrefixes defines the list of prefixes that the fuzzer will use to determine whether a given function is a property test or not |
| testPrefixes | prefix | |
| - | shrinkLimit | not implemented in Medusa |
| (?) | contractAddr | not found in Medusa (?) |
| (?) | propMaxGas | (?) |
| - | solcArgs | not implemented in Medusa |
| platformConfig -> crytic-compile -> args | cryticArgs | |
| - | quiet | not implemented in Medusa |
| - | dashboard | not implemented in Medusa |
| - | format | not implemented in Medusa |
| - | balanceContract | not implemented in Medusa |
| - | balanceAddr | not implemented in Medusa |
| - | maxValue | not implemented in Medusa |
| - | testDestruction | not implemented in Medusa |
| - | deployBytecodes | not implemented in Medusa |
| - | deployContracts | Addresses and contract names to deploy using the available source code. The deployer address is the same as the contract to test. Echidna will error if the deployment fails. (?) |
Compilation Configuration - it's not a one-to-one comparison because medusa implements it in a different way (platformConfig for each platform - crytic-compile/solc/truffle).
I'm currently running into Medusa missing the equivalent of deployContracts. It would be nice if this one could get some priority :slightly_smiling_face:
Hey @rappie - agreed, worth having this capability. We have a blocker to implement this feature. You can see it here
