[Bug-Candidate]: Missing coverage report information on mixed foundry/hardhat project

[aviggiano]

Describe the issue:

I am using echidna on a Liquity fork, which was originally a hardhat project, but which now has additional foundry tests.

After adding @crytic/properties as a foundry dependency (forge install crytic/properties --no-commit), I am able to see the lines hit by echidna on the coverage report. However, when I add the dependency as a hardhat module, I am not (yarn add --dev https://github.com/crytic/properties.git).

Sorry if this issue is not very well described, but the repository is closed source. I can try to replicate the problem later with an MWE, but the large codebase makes the setup a bit tricky, so I am not even sure if this bug is related to mixing foundry & hardhat

Code example to reproduce the issue:

@crytic/properties as a foundry dependency

cryticArgs: ["--solc-remaps", "@crytic/properties/=lib/properties/"]

@crytic/properties as a hardhat dependency

cryticArgs: ["--solc-remaps", "@crytic/properties/=node_modules/@crytic/properties/"]

Version:

Echidna 2.2.0 slither 0.9.2

Relevant log output:

No response

[gustavo-grieco]

Hi Antonio!,

Can you try reproducing this issue using the original liquity code? Perhaps that will be enough.

[elopez]

Hi @aviggiano! Are you targeting standalone .sol files or using hardhat/foundry to handle the compilation process (i.e. targeting . or a folder)? I see you mention using solc-remaps but I believe that option does not have any effect if you're using a compilation framework.

If you're targeting a folder and both hardhat and foundry are detected, crytic-compile will prefer foundry, which aligns with your observations that it only works when adding it to foundry 🤔 It should be possible to add --compile-force-framework foundry to cryticArgs to select which one is used during compilation, e.g.

cryticArgs: ["--compile-force-framework", "foundry"]

[aviggiano]

Hey @ggrieco-tob @elopez thanks for responding.

@elopez I believe your analysis is correct but I still can't make it work.

Initially, I was running the following command:

echidna contracts/TestContracts/invariants/echidna/EchidnaTester.sol --test-mode assertion --contract EchidnaTester --config config.yaml | tee /dev/null

And when I added --compile-force-framework to foundry, it errored:

[2023-08-04 13:40:43.92] Compiling contracts/TestContracts/invariants/echidna/EchidnaTester.sol... Done! (0.236543s)
echidna: Couldn't compile given file
stdout:
stderr:
INFO:CryticCompile:'forge clean' running (wd: /contracts/TestContracts/invariants/echidna/EchidnaTester.sol)
ERROR:CryticCompile:OS error executing:
Traceback (most recent call last):
  File "/Library/Python/3.10/lib/python/site-packages/crytic_compile/utils/subprocess.py", line 48, in run
    return subprocess.run(
  File "/opt/homebrew/Cellar/[email protected]/3.10.12_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 503, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/opt/homebrew/Cellar/[email protected]/3.10.12_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/opt/homebrew/Cellar/[email protected]/3.10.12_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 1863, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
NotADirectoryError: [Errno 20] Not a directory: PosixPath('/contracts/TestContracts/invariants/echidna/EchidnaTester.sol')
INFO:CryticCompile:'forge build --build-info --force' running
Traceback (most recent call last):
  File "/Library/Python/3.10/bin/crytic-compile", line 8, in <module>
    sys.exit(main())
  File "/Library/Python/3.10/lib/python/site-packages/crytic_compile/__main__.py", line 193, in main
    compilations = compile_all(**vars(args))
  File "/Library/Python/3.10/lib/python/site-packages/crytic_compile/crytic_compile.py", line 620, in compile_all
    compilations.append(CryticCompile(target, **kwargs))
  File "/Library/Python/3.10/lib/python/site-packages/crytic_compile/crytic_compile.py", line 110, in __init__
    self._compile(**kwargs)
  File "/Library/Python/3.10/lib/python/site-packages/crytic_compile/crytic_compile.py", line 530, in _compile
    self._platform.compile(self, **kwargs)
  File "/Library/Python/3.10/lib/python/site-packages/crytic_compile/platform/foundry.py", line 66, in compile
    with subprocess.Popen(
  File "/opt/homebrew/Cellar/[email protected]/3.10.12_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/opt/homebrew/Cellar/[email protected]/3.10.12_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 1863, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
NotADirectoryError: [Errno 20] Not a directory: 'contracts/TestContracts/invariants/echidna/EchidnaTester.sol'

After changing the compilation target to ., it did not error, but the coverage report was still not showing any lines.

echidna . --test-mode assertion --contract EchidnaTester --config config.yaml | tee /dev/null

Then, I've left the compilation target as . but changed the --compile-force-framework to hardhat, and it complained that you cannot use npx hardhat anymore:

[2023-08-04 13:47:24.71] Compiling .... Done! (1.745922s)
echidna: Couldn't compile given file
stdout:
stderr:
INFO:CryticCompile:'npx hardhat clean' running (wd: /contracts)
ERROR:CryticCompile:'npx' returned non-zero exit code 1
ERROR:CryticCompile:npm WARN ignoring workspace config at /contracts/.npmrc
stderr: Error HH12: Trying to use a non-local installation of Hardhat, which is not supported.
stderr: Please install Hardhat locally using npm or Yarn, and try again.
stderr: For more info go to https://hardhat.org/HH12 or run Hardhat with --show-stack-traces
INFO:CryticCompile:'npx hardhat clean --global' running (wd: /contracts)
ERROR:CryticCompile:'npx' returned non-zero exit code 1
ERROR:CryticCompile:npm WARN ignoring workspace config at /contracts/.npmrc
stderr: Error HH12: Trying to use a non-local installation of Hardhat, which is not supported.
stderr: Please install Hardhat locally using npm or Yarn, and try again.
stderr: For more info go to https://hardhat.org/HH12 or run Hardhat with --show-stack-traces
INFO:CryticCompile:Problem executing hardhat: npm WARN ignoring workspace config at /contracts/.npmrc
Error HH12: Trying to use a non-local installation of Hardhat, which is not supported.
Please install Hardhat locally using npm or Yarn, and try again.
For more info go to https://hardhat.org/HH12 or run Hardhat with --show-stack-traces

INFO:CryticCompile:'npx hardhat compile --force' running
INFO:CryticCompile:

ERROR:CryticCompile:npm WARN ignoring workspace config at /contracts/.npmrc
Error HH12: Trying to use a non-local installation of Hardhat, which is not supported.
Please install Hardhat locally using npm or Yarn, and try again.
For more info go to https://hardhat.org/HH12 or run Hardhat with --show-stack-traces

Traceback (most recent call last):
  File "/Library/Python/3.10/bin/crytic-compile", line 8, in <module>
    sys.exit(main())
  File "/Library/Python/3.10/lib/python/site-packages/crytic_compile/__main__.py", line 193, in main
    compilations = compile_all(**vars(args))
  File "/Library/Python/3.10/lib/python/site-packages/crytic_compile/crytic_compile.py", line 620, in compile_all
    compilations.append(CryticCompile(target, **kwargs))
  File "/Library/Python/3.10/lib/python/site-packages/crytic_compile/crytic_compile.py", line 110, in __init__
    self._compile(**kwargs)
  File "/Library/Python/3.10/lib/python/site-packages/crytic_compile/crytic_compile.py", line 530, in _compile
    self._platform.compile(self, **kwargs)
  File "/Library/Python/3.10/lib/python/site-packages/crytic_compile/platform/hardhat.py", line 195, in compile
    hardhat_like_parsing(crytic_compile, self._target, build_directory, hardhat_working_dir)
  File "/Library/Python/3.10/lib/python/site-packages/crytic_compile/platform/hardhat.py", line 49, in hardhat_like_parsing
    os.listdir(build_directory), key=lambda x: os.path.getmtime(Path(build_directory, x))
FileNotFoundError: [Errno 2] No such file or directory: 'artifacts/build-info'

[elopez]

Hi! I'll reply inline

Initially, I was running the following command:

echidna contracts/TestContracts/invariants/echidna/EchidnaTester.sol --test-mode assertion --contract EchidnaTester --config config.yaml | tee /dev/null

Okay, so this targets a .sol file, which is only supported by the solc platform (i.e. invoking solc directly) in crytic-compile. Forcing other platforms is expected to fail, as you saw below. Specifying solc remaps as you had on your first post would have a positive effect here.

And when I added --compile-force-framework to foundry, it errored:

(...)
"/opt/homebrew/Cellar/[email protected]/3.10.12_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 1863, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
NotADirectoryError: [Errno 20] Not a directory: PosixPath('/contracts/TestContracts/invariants/echidna/EchidnaTester.sol')
(...)

After changing the compilation target to ., it did not error, but the coverage report was still not showing any lines.

echidna . --test-mode assertion --contract EchidnaTester --config config.yaml | tee /dev/null

This targets the directory, so Foundry would be used if it's detected. Did echidna print any errors/warnings?

Then, I've left the compilation target as . but changed the --compile-force-framework to hardhat, and it complained that you cannot use npx hardhat anymore:

[2023-08-04 13:47:24.71] Compiling .... Done! (1.745922s)
echidna: Couldn't compile given file
stdout:
stderr:
INFO:CryticCompile:'npx hardhat clean' running (wd: /contracts)
ERROR:CryticCompile:'npx' returned non-zero exit code 1
ERROR:CryticCompile:npm WARN ignoring workspace config at /contracts/.npmrc
stderr: Error HH12: Trying to use a non-local installation of Hardhat, which is not supported.
stderr: Please install Hardhat locally using npm or Yarn, and try again.
stderr: For more info go to https://hardhat.org/HH12 or run Hardhat with --show-stack-traces
(...)

Do you have a local node_modules with a hardhat installation on that directory? it seems npx is running a global installation of hardhat, likely because there's not a node_modules with hardhat around your current folder. You can try to yarn add --dev hardhat and see if that makes a difference.

Do you have a monorepo or non-standard directory set up on your project? I know you mentioned the repo is private but if you could outline the folder structure you have and where key files are (e.g. package.json, hardhat.config.js, foundry.toml, node_modules, solidity contracts) that might help.

[aviggiano]

hey

Sorry for taking so long to reply. Since this isn't a big problem, as I avoided it by using foundry's git submodules instead of hardhat's node modules, I eventually moved on to other stuff.

Indeed this is a monorepo. The project is a liquity fork, which uses yarn workspaces, but the team added new features using forge. There is a node_modules folder inside packages/contracts/, just as liquity, but also now a new foundry.toml file with the Solidity files inside packages/contracts/contracts

I will eventually need to get back at this issue before merging my echidna pull request, so I'll try your suggestion later.