Testing Framework Improvements

[SupunS]

Issue To Be Solved

https://github.com/onflow/cadence/issues/331 proposed a testing framework for cadence. An initial version of this proposal is now implemented, and supports basic functionalities to write tests.

However, there is still some room for improvement, and not all functionalities suggested in https://github.com/onflow/cadence/issues/331 are implemented.

Below is a set of improvements remaining to be done:

Assertions

• [x] #1959 fun fail(message: string) Currently there's a fail() function that takes no arguments. Remove the existing implementation, and implement this natively, to accept an optional 'message' argument. Since the argument is optional, this is a backward compatible change

Matchers

https://github.com/onflow/cadence/pull/1863 laid the foundation for implementing matches. It also added equal() matcher that can be used to test the equality of values.

The proposal suggests some more convenient functions for creating most commonly used matchers. Many of these might be able to implement in cadence (some may have to be implemented natively).

• Matcher functions:
  • [ ] fun haveElementCount(_ count: Int): Matcher: returns a matcher that succeeds if the tested value is an array or dictionary, and has the given number of elements
  • [ ] fun beEmpty(): Matcher: returns a matcher that succeeds if the tested value is an array or dictionary, and the tested value contains no elements
  • [ ] fun beNil(): Matcher: returns a matcher that succeeds if the tested value is nil
  • [ ] fun contain(_ element: AnyStruct): Matcher: returns a matcher that succeeds if the tested value is an array that contains a value that is equal to the given value, or the tested value is a dictionary that contains an entry where the value is equal to the given value
  • [ ] fun beGreaterThan(_ value: Number): Matcher returns a matcher that succeeds if the tested value is a number and greater than the given number
  • [ ] fun beLessThan(_ value: Number): Matcher returns a matcher that succeeds if the tested value is a number and less than the given number
  • [ ] fun beSucceeded(): Matcher returns a matcher that succeeds if the tested value is a transaction result or a script result, and the result has a succeeded status
  • [ ] fun beFailed(): Matcher returns a matcher that succeeds if the tested value is a transaction result or script result, and the result has a failed status
• Built-in matcher combinators:
  • [ ] fun not(matcher: Matcher): Matcher: returns a matcher that negates the given matcher

Unit Testing

Currently, it is possible to directly import a contract and invoke contract functions within the test cases.

It is also very useful to extend this first-class-ness to script and transactions as well.

• [ ] Allow importing a script/transaction and executing it directly from test cases
• [ ] Allow creating accounts from test cases.

More ideas are here: https://github.com/onflow/cadence/issues/1993

Integration Testing

https://github.com/onflow/cadence/pull/1868 added support for deploying a contract to a blockchain, and https://github.com/onflow/cadence/pull/1849 added support for running transactions (or scripts) that import/use the deployed contract.

One limitation of the current implementation is that, to test a contract, developers always have to submit a transaction/script, and perform the assertion in the script/transaction. One way to improve the DevX is to return an object representing the contract as the result of contract deployment (i.e: deployContract function).

This object would act as a stub/proxy for calling methods of the deployed contract. Then instead of having to write transactions/scripts and submitting them to test a contract, developers can directly test functions/fields of the contract within the test cases. Reduces a lot of boilerplate code

• [ ] Make deployContract function return an object that represents the deployed contract. e.g: fun deployContract<T>(...): T
  • Implementation: Under the hood,
    • convert the function-call/field-access to a script that does the same
    • execute the script against the blockchain (same as executeScript())
    • return the result/value of the script as the result of the function-call/field-access
• [ ] Add fun Blockchain.getContract<T>(at address: Address): T
  • Returns the instance of the contract deployed at the given address
  • Implementation: proxy which forwards function calls, and field access through scripts

Deployed contracts:

• [ ] Make core contracts available by default. e.g: NonFungibleToken, etc.

Open questions:

• How to decode values, because certain types might only be available on the chain, and the test script may not have type info.

Logs & Events

• [ ] Provide a way to collect/retrieve logs from the blockchain
  • fun Blockchain.logs(): [String] Returns all logs up to that point.
  • Another option for advanced use-cases is to provide a way to set a log listener. e.g: fun Blockchain.setLogger(_ logger: ((String): Void))
• [ ] Provide a way to collect/listen to events emitted

Other

• Property-based testing. Maybe use and expose https://github.com/flyingmutant/rapid

### Tasks
- [ ] https://github.com/onflow/cadence-tools/issues/103
- [ ] https://github.com/onflow/cadence-tools/issues/98 

[m-Peter]

@SupunS Could you check the items from the Matchers section, as completed from https://github.com/onflow/cadence/pull/2420 ? And maybe assign the issue to me :pray: (as we are working on it as part of the grant proposal).

cc @chrisackermann @turbolent

[SupunS]

Hey @m-Peter! Thanks for the great contributions on the Matchers and for improving the test-framework 🎉 . And yeah, the currently implemented set of Matchers completes the list specified here. I think that should be sufficient for the majority of use-cases.

[SupunS]

All the functionalities specified in the description are now supported (Thanks to awesome work by @m-Peter!). Hence closing this issue. Please feel free to open new issues for any new improvement.