Add "simulate" option to the transaction endpoint

[algoanne]

Problem

dryrun and tealdbg are seriously limited because they don't use the real BlockEvaluator. This means if I debug a transaction group, that, for example, has a pay in the first transaction and an app call in the second transaction, the balances of the accounts involved in the pay won't reflect the pay when they get to the app call because each txn is evaluated "by hand" independently. Non-appl txns without code aren't even evaluated! This makes our debugger useless for a large fraction of use cases.

Solution

Dry run should be refactored until it can evaluate the entire txgroup provided by invoking a standard BlockEvaluator, rather than the ad-hoc loops in tealdbg/local.go. That code almost gets this right - it creates an apply.Balances using makeBlancesAdapter so that transactions can be processed against it. But this approach needs to be moved a level up.

We need a single ledger/balance object that is used throughout the evaluation. That can be accomplished by creating a specific simulation evaluator that uses an appropriate ledger and uses StartEvaluator to create a BlockEvaluator for use on the whole group.

As part of this story, provide a new algod endpoint to simulate the submission and execution of a txn group without actually committing the result to the blockchain.

This should be a new option to the transaction submission endpoint called "simulate". For example: "goal clerk simulate".

We are not touching existing dryrun endpoints (these will become deprecated eventually).

As part of this story, we also want to provide a new output: the stack trace detailing every step of the evaluation. This trace is output as a (compressed) JSON file. Details of trace: transaction #, PC #, changes to Scratch space, the stack, and changes to local/global/box state.

Out of scope follow-ups:

• setting state directly
• giving unlimited access to be able to estimate foreign arrays args and fees.
• adding the TEAL line and PyTeal line to the trace (from some CLI or something).

[tzaffi]

@michaeldiamant had some good links to EVM tracing features which may serve as useful references:

• https://geth.ethereum.org/docs/dapp/custom-tracer
• https://geth.ethereum.org/docs/dapp/tracing

[jeapostrophe]

I interpret this story as being mostly about the debugging process, e.g. the stack trace and so on.

I believe that the more important use of this will be for end-user applications that want to learn about on-chain without paying for it. For example, there is an AMM and I want to find out the current swap ratio.

The status quo on Algorand is for the developer of the AMM to either (a) provide an SDK that interprets the application state after communicating with the node/indexer or (b) document the format thereof and let clients do it themselves. Reach automatically provides the first for all Reach programs that use the View feature.

The status quo on Ethereum is that a contract will have an API method that is marked as "pure" or "view" and then the client (e.g. the actual Web browser looking at UniSwap) will call the node with a request that is roughly "Please run this function and give me the result, based on your current ledger, and don't charge any money". This is how Ethereum already works and it has led to an easy-to-use and robust development environment. Reach automatically provides these ABI functions for all Reach programs that use the View feature, but they aren't very useful without the simulation system.

Translating the "Ethereum way" into Algorand parlance... we just need the ability to run an appl txn, with the node's ledger state, and see the logs that come out to interpret the result.

I fear that approaching this problem from too much of a debugging/you-can-do-everything will make the feature more expensive to provide, so that node providers won't enable it, AND more expensive for you (the wonderful team at Algorand) to actually implement. I like the debugging feature, but I feel like there's something a bit simpler that is wanted.

[jeapostrophe]

Unrelatedly, something that I thought was meant by #2520 was an on-chain method of calling such functions in a cheaper way. I believe that I pay a fee and make an inner txn for calling another application, because its state may change. But, if we know that I am just reading its data, then presumably this can be cheaper. Again, this is how things work on Ethereum with view and pure function, although not specifically because the network knows about those ideas, but because they actually cost much much less gas (the most expensive thing that Ethereum programs do is record state changes). I like the scoping of this issue, but I think you should make a separate one for a better way to do it on-chain (or just say... It will never be cheaper! You are really paying for memory bandwidth! --- I don't like that answer, because I don't believe it... the outer tin is paying for including the inner application in its app array.)

[jannotti]

Just adding a note that we will probably have to do some explicit work to handle the Fees and SIgnatures, as the BlockEvaluator assumes they have already been checked.