Cadence testing framework

[turbolent]

Issue To Be Solved

It would be nice to write tests for Cadence in Cadence.

Suggest A Solution

Once the Go testing library is ready, it could be exposed to Cadence

[turbolent]

Here's a first proposal for the framework:

Cadence Testing Framework

Assertions/Expectations/Matchers

• fun assert(_ condition: Bool, message: String):

  • Accepts boolean expression
  • Simple implementation: Abort test instead of aborting whole program
  • Improved implementation: Could analyze expression, like Scala Test's assert macro
    • Translate expression to a matcher

• fun fail(message: string):

  • Aborts the current test
  • Message argument is optional

• fun expect(_ value: AnyStruct): Expectation:

  • Returns an expectation for the given value
  • Open question: how could we support resources, e.g. expect(optionalResource).to(not(beNil()))

• Expectation

  • An expectation is an object that allows a value to be tested against a matcher

    pub struct Expectation {
    
        let value: AnyStruct
    
        pub init(value: AnyStruct) {
            self.value = value
        }
    
        pub fun to(_ matcher: Matcher) {
            if !matcher.test(self.value) {
                fail()
            }
        }
    }
    

• Matcher

  • A matcher is an object that consists of a test function and associated utility functionality

    pub struct Matcher {
    
        pub let test: ((value: AnyStruct): Bool)
    
        pub init(test: ((value: AnyStruct): Bool)) {
            self.test = test
        }
    
        /// Combine this matcher with the given matcher.
        /// Returns a new matcher that succeeds if this and the given matcher succeed
    
        pub fun and(_ other: Matcher): Matcher {
            return Matcher(test: fun (value: AnyStruct): Bool {
                return self.test(value) && other.test(value)
            })
        }
    
        /// Combine this matcher with the given matcher.
        /// Returns a new matcher that succeeds if this and the given matcher succeed
    
        pub fun or(_ other: Matcher): Matcher {
            return Matcher(test: fun (value: AnyStruct): Bool {
                return self.test(value) || other.test(value)
            })
        }
    }
    

  • Built-in matcher functions:

    • fun equal(_ value): Matcher: returns a matcher that succeeds if the tested value is equal to the given value
    • fun haveElementCount(_ count: Int): Matcher: returns a matcher that succeeds if the tested value is an array or dictionary, and has the given number of elements
    • fun beEmpty(): Matcher: returns a matcher that succeeds if the tested value is an array or dictionary, and the tested value contains no elements
    • fun beNil(): Matcher: returns a matcher that succeeds if the tested value is nil
    • fun contain(_ element: AnyStruct): Matcher: returns a matcher that succeeds if the tested value is an array that contains a value that is equal to the given value, or the tested value is a dictionary that contains an entry where the value is equal to the given value
    • fun beGreaterThan(_ value: Number): Matcher returns a matcher that succeeds if the tested value is a number and greater than the given number
    • fun beLessThan(_ value: Number): Matcher returns a matcher that succeeds if the tested value is a number and less than the given number

  • Built-in matcher combinators:

    • fun not(matcher: Matcher): Matcher: returns a matcher that negates the given matcher

Blockchain

• A blockchain is an environment to which transactions can be submitted to, and against which scripts can be run.

  pub struct Blockchain {
  
      pub let backend: BlockchainBackend
  
      init(backend: BlockchainBackend) {
          self.backend = backend
      }
  
      pub fun executeScript<Value>(_ script: String): ScriptResult<Value> {
          return self.backend.executeScript(script)
      }
  
      pub fun addTransaction(_ transaction: Transaction) {
          self.backend.addTransaction(transaction)
      }
  
      /// Executes the next transaction, if any.
      /// Returns the result of the transaction, or nil if no transaction was scheduled.
      ///
      pub fun executeNextTransaction(): TransactionResult? {
          return self.backend.executeNextTransaction()
      }
  
      pub fun commitBlock() {
          self.backend.commitBlock()
      }
  
      pub fun executeTransaction(_ transaction: Transaction): TransactionResult {
          self.addTransaction(transaction)
          let result = self.executeNextTransaction()!
          self.commitBlock()
          return result
      }
  
      pub fun executeTransactions(_ transactions: [Transaction]): [TransactionResult] {
          for transaction in transactions {
              self.addTransaction(transaction)
          }
          var results: [TransactionResult] = []
          for transaction in transactions {
              let result = self.executeNextTransaction()!
              results.append(result)
          }
          self.commitBlock()
          return results
      }
  }
  
  pub enum ResultStatus {
      case succeeded
      case failed
  }
  
  pub struct TransactionResult {
      pub let status: ResultStatus
  }
  
  pub struct ScriptResult {
      pub let status: ResultStatus
  }
  
  pub struct interface BlockchainBackend {
  
      fun addTransaction(_ transaction: Transaction)
  
      fun executeNextTransaction(): TransactionResult?
  
      fun commitBlock()
  
      fun executeScript<Value>(_ script: String): ScriptResult<Value>
  }
  

• fun newEmulatorBlockchain(): Blockchain returns a blockchain which is backed by a new emulator instance

• fun beSucceeded(): Matcher returns a matcher that succeeds if the tested value is a transaction result or a script result, and the result has a succeeded status

• fun beFailed(): Matcher returns a matcher that succeeds if the tested value is a transaction result or script result, and the result has a failed status

Contracts

Unit Testing

For unit testing, contracts can be imported from files. The import declares the contract constructor, instead of singleton object like on-chain. This allows instantiating the contract without deploying to a blockchain.

Integration Testing

For integration testing, allow the deployment of contracts to a blockchain

• fun Blockchain.deployContract<T>(to address: Address, ... initializerArguments): T

  • Deploys the contract to the given address and returns an instance
  • Implementation: proxy which forwards function calls and field access through scripts

• fun Blockchain.getContract<T>(at address: Address): T

  • Returns the instance of the contract deployed at the given address
  • Implementation: proxy which forwards function calls, and field access through scripts

Open Questions

• How are address imports handled when contract files are imported?

[MaxStalker]

Looks promising! 👍 Well done, Bastian!

For deployment - sometimes you want to have a custom deployment script, like we have in one of the core contracts. Where we do some additional logic before setting account code. Is there a way to support this? Or this could be handled by submitting a transaction?

Q: How the address imports handled when contract files are imported A: In my JS solution I was using an object/mapping, which would replace addresses before deployment. We can do something like fun Blockchain.deployContract<T>(to address: Address, imports: {String: String}, ... initializerArguments): T as well and tell devs they shall do it manually? 🤔

[MaxStalker]

Also, we need a mechanism to catch and process events. Maybe we can put them into transaction/script result, so we won't need to search for them by block height?

[turbolent]

@MaxStalker thanks for the feedback!

custom deployment script, like we have in one of the core contracts.

Would you be able to point me to them?

Re: address imports: yeah, that's an option. Do you think that would get quite repetitive an unmaintainable quickly? I guess the imports argument could be a global to avoid repetition in multiple tests

[cybercent]

@turbolent nice work! Could convention over configuration work for contract addresses? Like including the address in the contract file name: 0x01.NbaTS.cdc

[rheaplex]

How are addresses and signatures handled within this?

Could scripts create a chain, deploy contracts to it (with or without a custom deploy script), then send transactions to it to test it? Scripts could be deployment/test systems then. If they could include each other this would be nice and modular.

Address resolution is a broader problem, and I'm wary of recreating the C preprocessor, but specifying import X from "<FILEPATH>" could be replaced by the compiler once the address that FILEPATH is deployed to is known.

[turbolent]

More inspiration for functionality: https://github.com/gakonst/foundry/tree/master/forge, e.g. mocking environment information, like timestamp, block number, etc.

[SupunS]

I've been looking at this over the last couple of days, and one question I have, that hasn't seemed to be discussed here yet is, 'Where/how do tests run?' If I explain it a bit more:

There are two things that would 'run' in a test.

• The test case (test script) - The test itself is written in Cadence. Thus, it would need to run against a backend/environment because Cadence alone doesn't provide some functionalities.
  • e.g: Creating a public key value for test assertions - Public key validation etc. is provided by the environment (fvm)
• A script/transaction that is run within a test
  • e.g: creating a blockchain backend with newEmulatorBlockchain() and running a script/transaction against it with executeScript("some script")

In the second case, newEmulatorBlockchain could start an emulator and run the subsequent requests on it. This looks straightforward.

My question is more on how do we run the test script itself (first case)? Couple of options I could think of:

• Run it with a mocked environment - like a sandbox with limited functionality. The downside is that, it's probably a lot of work to implement the mocked environment (compared to option-II). Also may be hard to keep the consistency with the actual networks. Might not be the best solution
• Start the emulator instance and run the ‘test script’ against that. The downside is that there will be two emulator instances (two environments/sandboxes): one for the test-script and one (or more) for the blockchain(s) started within the test.

[SupunS]

btw, I'm thinking of a cadence test script to be something similar to the one below:

// A 'setup' function that will always run before the rest of the methods.
// Can be used to initialize things that would be used across the test cases.
// e.g: initialling a blockchain backend, initializing a contract, etc.  
pub fun setup() {
}

// Test functions that start with the 'test' prefix

pub fun testSomething() {
}

pub fun testAnotherThing() {
}

pub fun testMoreThings() {
}

// A 'tearDown' function that will always run at the end of all test cases.
// e.g: Can be used to stop the blockchain back-end used for tests, etc. or any cleanup
pub fun tearDown() {
}

[turbolent]

@SupunS regarding "Where/how do tests run?": I think it makes sense to re-use/piggy-back on the existing environment we have, instead of re-implementing most of what Cadence needs (runtime.Interface).

The "lowest" level of the implementation would be FVM: It implements all that Cadence needs and should be sufficient to implement the test runner for running the tests. If we were to build on FVM, we could integrate it e.g into the CLI.

It could also be implemented in the Emulator (which builds on FVM), but from what I can see, we do not need any of the functionality of the Emulator for the tests themselves (we of course want the Emulator for what is tested: sending transactions, scripts, etc.)

It might be worth investigating the FVM option first and seeing if it is sufficient. If it is not, we can still take the code and port it to the Emulator.

[bluesign]

I love this initiative, but to be honest this feels totally wrong to me.

Does javascript initiate JS interpreter when writing tests? or golang evals code ? If we gonna make a cadence testing framework to test cadence, isn't it wrong to expose a blockchain ?

We have script context without mutability, but it can emulate all necessary blockchain features. It has some issues with events ( which I am pushing to fix https://github.com/onflow/flow-go/issues/2813 ) but other than that, isn't script / transaction just a function ?

Btw I saw @SupunS comment:

The test case (test script) - The test itself is written in Cadence. Thus, it would need to run against a backend/environment because Cadence alone doesn't provide some functionalities. e.g: Creating a public key value for test assertions - Public key validation etc. is provided by the environment (fvm)

I think we never test for key validation. Not with js-testing ( which basically creates account and manages, or with any other testing framework like overflow) we only test, different account / different role

Run it with a mocked environment - like a sandbox with limited functionality. The downside is that, it's probably a lot of work to implement the mocked environment (compared to option-II). Also may be hard to keep the consistency with the actual networks. Might not be the best solution.

Here Mocked environment has only few things to mock, like getBlock etc for timestamps. But mocking is usually responsibility of the coder, not the test platform. ( Like Clock example from FIND contracts, it can be mocked when user wants to mock something ) Ofc we can provide ready made mocked objects, but using them should be not our responsibility.

I think best option is make a Testing contract, with some generic helper functions ( like js-testing has ) such as :

fun getAccount(name: String): AuthAccount {
   if let account =  self.accounts[name]{
      return getAuthAccount(account)
   }
   self.accounts[name] = AuthAccount(payer: self.account).address
   return getAuthAccount(self.accounts[name])
}

Only thing needed is special handling on panics, which can be made possible as we will have Testing contract running as a native code. ( Testing can have runTransaction, executeScript functions which can catch the panics )

[SupunS]

There are a few things I would want to clarify. (Disclaimer: I'm not very familiar with JS, so I'm gonna use Java/JVM for examples. But the idea is the same :) )

• Environment: Here the environment is similar to the JVM in java / Go runtime / JS engine. Just like the JVM is needed to run any java code, an environment is needed to run any cadence code, including tests. The implementation of the environment is not something that needs to be exposed to the user. There are fundamental operations handled by the environment (for e.g: handling storage, value encoding/decoding, etc.), and we don't want to expose these to the user, or let users mock these, because it could mess up cadence fundamentals. Just like we don't mock the JVM in the Java world.

  However, having said that, there may be cases where mocking some environment functionalities could be useful. This initial version of the test framework doesn't support mocking. But we can add mocking of a subset of functionalities in future as needed. For e.g: allow the dev to "register" a mocked object during test "setup()", and use that during execution.

• Unit testing: I think this is what you were referring to. This is the simple scenario. Very Similar to writing a unit test in JS/Java/Go. Everything runs on the same environment. You don't need a blockchain for this. Can simply import a contract, and test its functionalities.

  This proposal doesn't suggest a way to directly run a script/transaction. That may be a good thing to add for unit testing. But I don't know how we could import an already written transaction or a script to the test. i.e: How to say run this X script. Sure we can figure out a way.

• Integration testing: Integration testing allows you to start up a blockchain instance and run scripts/Tx against it. I would say, in the java world, this is something similar to starting up a micro-service in a separate JVM, and sending CRUD requests against it, over a network/protocol.

[bluesign]

thanks @SupunS

My main question is why do we need new blockchain for integration testing? We are already running on some blockchain, can't we just use it for integration testing?

I mean Transaction / Scripts are functions technically, accounts we can create on the fly. Asserting failures and panics seems will be possible.

[SupunS]

Sorry, I didn't get what you mean by "We are already running on some blockchain". Do you mean the environment where the test itself is running? i.e: sharing the environments for the test case and the code that is being tested?

[bluesign]

Start the emulator instance and run the ‘test script’ against that. The downside is that there will be two emulator instances (two environments/sandboxes): one for the test-script and one (or more) for the blockchain(s) started within the test.

I was referring to this, 2 environments running ( 2 emulator instances ) .

sharing the environments for the test case and the code that is being tested

yes exactly this one, if we will run tests In emulator instance, we already have a blockchain, we don't need to create a new one.

[SupunS]

Running the tests on an emulator was one option. But with what Bastian suggested, we don't really need an emulator there, we can rely on the FVM to provide the functionality needed for cadence (emulator is FVM + more). So the tests would run sort of locally with minimal requirements (i.e: a very light implementation of the environment).

About sharing the environment, I favour having them separately, because:

• Can write isolated tests. Gives the ability to write tests such that, one test/test-suite wouldn't affect the rest of the tests/test-suites, by maintaining one backend per test/test-suite. If they want to share the state between tests, they can do that too, by using one backend for all tests. So it's flexible.
• Running tests requires additional functionalities, like importing test stdlib, maybe loading contracts/transactions from files, etc. But the code that is being tested doesn't need those.
• There can be different backends. For example, in the initial version, we would support emulator-backend as the blockchain. In future, there can also be local-net etc. backends as well.

[bluesign]

@SupunS yeah actually benefits are real. Thanks for explaining in detail.

Btw will there be any work related to cleanup environment e.g. Cadence/runtime coupling ? I mentioned that before with some issue ( https://github.com/onflow/cadence/issues/1160 ) Currently Cadence / FVM have very tight coupling with Runtime interface. Which causes a lot of problems, for example:

• runtime interface changes require all consumers of cadence api to update code )
• Cadence has to know about script / transaction environments ( and now testing probably )

Maybe I am totally missing something but I think Cadence being a language and shifting some Environment specific functions to FVM side can be better.