Node test runner outputs meaningless error messages

[ItsNickBarry]

Version of Hardhat

3.0.0-next.4

What happened?

The Hardhat node test runner outputs meaningless information on some failing tests. Two examples:

1. (edit: this issue has been fixed) assert.match says that I'm comparing two different types of values.

   AssertionError: The input did not match the regular expression /nomatch/. Input:
   
   '// SPDX-License-Identifier: MIT\n' +
     'pragma solidity *;\n' +
     '\n' +
     "import 'hardhat/console.sol';\n" +
     '\n' +
     'contract ContractWithLogs {\n' +
     '    function fn() external pure {\n' +
     "        console.log('log1');\n" +
     "        console.log('log2');\n" +
     '    }\n' +
     '}\n'
   
     Comparing two different types of values. Expected regexp but received string.

2. assert prints parts of the test code which are unrelated to the error.

   AssertionError: The expression evaluated to a falsy value:
   
     readContractSource('ContractWithoutLicense')
   
   - Expected
   + Received
   
   - true
   + false

In both cases the failures are expected, but the error messages are incorrect.

Minimal reproduction steps

I've added failing test cases in two repositories.

Clone hardhat-log-remover and checkout 35bbc6e3bbb17cc73202d1722718d281343795f0. Run yarn run hardhat test.

Clone hardhat-spdx-license-identifier and checkout f785f0f1dee672909ca346848a944303d5531799. Run yarn run hardhat test. Try making other matchers fail to get other unexpected outputs.

Search terms

node test runner output

[galargh]

Hi! Could you please link to the test cases that produce the unexpected output? If it's applicable, would you mind also sharing what output you'd expect for these test cases?

[ItsNickBarry]

https://github.com/ItsNickBarry/hardhat-log-remover/commit/2a5c5ec66828b0188f73b86bd33bbcdc94959151

https://github.com/ItsNickBarry/hardhat-spdx-license-identifier/commit/a762d32a4e45afe9df9757cda8aff6ed524d179e

The first should not include this line: Comparing two different types of values. Expected regexp but received string.

The second should indicate this expression as the source of failure: !contentsBefore.includes(HEADER_BASE)

[galargh]

I was able to track down and fix the issue with regex matches 🥳 https://github.com/NomicFoundation/hardhat/pull/6658

As for the other one, I still haven't replicated it. Could you let me know what node version you're using?

[ItsNickBarry]

22.14.0

[galargh]

I was able to replicate the second issue in a smaller setting - https://github.com/galargh/tsx-esm-assert

It's a really weird one. It's even sensitive to the whitespace, variable name, and string literal changes in the test case 🤯 At the moment, I suspect it might have to do with the way tsx/esm is dealing with the code but I did not pinpoint it exactly.

[alcuadrado]

Note the same issue, but reminds me a bit to https://github.com/privatenumber/tsx/issues/548

[alcuadrado]

I could also reproduce it in 24.0.1.

I think we should both report it to tsx, and trim in the meantime trim the messages The expression evaluated to a falsy value: to The expression evaluated to a falsy value. It's less informative, but correct, as the stack trace is correctly pointing to the line with the assert.