Stackoverflow on DAML-LF with too much nesting

[cocreature]

The following DAML Script will produce a stackoverflow when loaded into sandbox (using the 1.1.0 snapshot but it’s not specific to that) https://gist.github.com/cocreature/d6b3d222dec9937d0590e0862eaf5591

This is not actually specific to DAML Script. The only somewhat surprising fact here is that a large do block in DAML Script produces nested DAML-LF but that is somewhat unavoidable. Even if you inline the typeclass methods you end up with a free monad which is nested.

While we could solve the issue in the decoder which is the part that is exploding here this only slightly bumps the limit until we run into the protobuf recursion limit.

@remyhaemmerle-da proposed the following steps which I support:

1. Define a nesting level for DAML-LF that matches the protobuf level. Anything with more nesting will be a hard error.
2. ~Ensure that the DAML-LF decoder, typechecker and so on do not stackoverflow on that level. This could be by trampolining or simply by making sure that we do not use that much stack up to this recursion level (note that this is somewhat tricky to control since the decoder is also a library so we don’t know how deep we are on the stack when we start)~ [Done: The decoder have been completly stack safe]
3. Ensure that the compiler produces at least a sensible error and ideally automatically outlines expressions that run over the limit so you do not run into this error.

Workaround in the meantime

Manually outline things, e.g., turn

a = do
  x 
  y
  z

into

a = do
  x
  b
b = do
  y
  z