A new world: Reimplementation of the off-chain environment

[Robbepop]

To make it short and precise: Our off-chain environment is not more than a first attempt to get off-chain testing for smart contracts. Its design and implementation has never been really good but also it was not strictly required for its purposes back then - the main goal was for it to be kind of working which it did.

However, times have changed and we definitely are in need of some heavily polished implementation of an off-chain environment. The main problems of our current off-chain environment are stemming from the fact that it is not at all close to simulate what the on-chain environment (a.k.a. the real thing) does. Instead it tries to be practical and combines certain entities if that's what made its implementation simpler.

The plan for this go to forward is the following:

• [ ] Discuss a new design for the off-chain environment that is more similar to the actual on-chain. Maybe it is even running an actual chain and another program is reading and interpreting the test commands from the tests? (Just thinking ...)
• [ ] Implement a proper interface between ink! and such an off-chain environment. We no longer want to have the entire implementation of it inside of ink_env. Instead we want to move out the implementation of such a new off-chain environment into its own crate, maybe even a crate that does no longer live inside the ink! repository so that it can be reused for other Rust based contract eDSLs?
• [ ] Implement the actual new off-chain environment.
• [ ] Link ink_env to the new off-chain environment.
• [ ] Design the new test API (our current test API is equally broken).
• [ ] Implement the new test API (in ink_env) that requires proper interfaces towards the off-chain environment. So the test API needs to be a proper part of the overall interface.
• [ ] Enable the event_must_have_unique_topics test in unique_topics.rs. This test had to be disabled since https://github.com/rust-lang/cargo/issues/6915 currently prevents crates from activating features in dev-dependencies of child crates.

[XSvr]

Link ink_env to the new off-chain environment. after long time time work，then invoke call back function。just like substrate off-chain #558 the reason of my question

[Robbepop]

Off-chain Redesign

Problems With Old Design

• Is the notion of an execution context actually a good idea?
  • Must always define "big object" and cannot set just a subset of the fields.
  • Current implementation does not sync state and execution context.
  • Proposal: Maybe better to split up the API and instead have smaller setters and getters such as:
    • next_call_caller
    • next_call_gas_limit
    • etc.
  • Before defining our own new test API:
    • Look what NEAR and its mothership object approach does.
• Data structures are split up into AccountsDB´ and ContractsDB` etc. instead of using a root database in the lowest level of abstraction just as Substrate does.
• Build interface on top of the lowest level SEAL API instead of the Backend trait API provided by ink_env.
  • Use interfaces found in ink/crates/env/engine/on-chain/ext.rs
• Build new Off-Chain environment in its very own crate.

Requirements

• For the time being ink! will integrate the new off-chain environment as a third option for the backend that is enabled when ink! is compiled with std and ink-experimental-engine crate features.
  • This allows us to test and use the new off-chain engine directly.
  • Problem: Both, old and new off-chain environment do not share a common test API.
  • Examples will be required to have new off-chain test using the new test API. Old tests are required to be feature flagged with #[cfg(not(feature = ink-experimental-engine))] for the time being.
  • When the new off-chain engine implementation is done we will entirely replace the old off-chain environment with the new one and remove all code associated just to the old engine.
• Implementation and design should focus on simplicity.
• The off-chain environment is expected to be compiled using the Rust standard library.
• Unlike the real contracts pallet there is no high performance, low lattency requirement, however computation must be deterministic.
• The off-chain environment implementation must not depend on any ink! crate.
  • For the MVP we create the new off-chain environment crate inside the ink! repository under crates/engine.
    • Maybe be creative and find a better name.
• Compilation time of the off-chain environment must be kept at a minimum.
  • The off-chain envrionment crate must be recompiled everytime a new contract is tested.
  • Ideally compilation shouldn't take more than a few seconds.
• Requirements are generally the same or at least extremely similar to the requirements on the real on-chain environment implementation.
• Chain Extension
  • It must be possible to register a new chain extension given a func_id and a closure that represents the emulated function body of the chain extension.
  • The function body of the emulated chain extension must be able to query, inspect and mutate the emulated runtime storage and information.
• MVP Functionality:
  • Example smart contracts for the MPV:
    • Flipper
    • Incremented
    • ERC-20
  • Required for the MVP:
    • seal_deposit_event
    • seal_set_storage
    • seal_get_storage
    • seal_clear_storage
    • seal_input
    • seal_return
    • seal_caller
    • seal_address
    • seal_balance
    • seal_value_transferred
    • seal_println
  • Not required for the MVP:
    • seal_now
    • seal_random
    • seal_transfer
    • Cross Calling:
      • seal_instantiate
      • seal_call
    • Chain Extension:
      • seal_call_chain_extension
    • Exotic Getters:
      • seal_block_number
      • seal_weight_to_fee
      • seal_gas_left
    • Storage Rent:
      • seal_terminate
      • seal_rent_allowance
      • seal_minimum_balance
      • seal_tombstone_deposit
      • seal_set_rent_allowance
      • seal_restore_to
    • Hash Functions:
      • seal_hash_keccak_256
      • seal_hash_blake2_256
      • seal_hash_blake2_128
      • seal_hash_sha2_256