VM callbacks can now raise (and be caught, or show their stacktrace)

[timotheecour]

before PR

VM callbacks can't be caught, so code behaves differently at RT vs CT if VM code calls a callback that raises. eg:

• catch doesn't work
• on crash, no useful context or stacktrace is shown so we have no idea how we got here

after PR

• VM callbacks can be caught, all that's needed is an extra arg registerCallback(.., mayRaise=true)
• the exception (whether caught in regular code or if there's no handler, when the program crashes with stacktrace), contains the relevant context
• stacktrace inside VM callback is also shown when nim itself is compiled with --stacktrace:on
• nimscript implementation of callbacks is now simplified thanks to this and shows more informative stacktraces, eg when nim is built with --stacktrace:on
• so things like walkDir(..,checkDir=true), writeFile, removeDir etc now work correctly in VM
• fixes https://github.com/nim-lang/Nim/issues/12676 (listFiles(..,checkDir=true) + friends now work; I did not change the default semantic to be consistent with walkDir)
• a new way to define internal procs is defined ({.internalproc.}), analog to {.compilerproc.}, but for semantic phase instead of codegen. This enables writing handlers in pure nim code instead of having to manually write AST; this not only simplifies writing compiler code, but it also allows change those handlers without having to recompile nim. A key aspect is that these are private (such symbols don't need to be exported).

example

proc fun1() = writeFile("/tmp/nonexistant/bar.txt", "foo")
proc fun2()=fun1()
static: fun2()
# note: try/catch would also work inside VM after this PR (unlike before PR)

before PR: no context shown, hard to debug in complex programs

/Users/timothee/git_clone/nim/Nim_devel/lib/system/io.nim(706) writeFile
Error: unhandled exception: cannot open: /tmp/nonexistant/bar.txt [IOError]

after PR:

stack trace: (most recent call last)
/Users/timothee/git_clone/nim/timn/tests/nim/all/t10375c.nim(93, 15) t10375c
/Users/timothee/git_clone/nim/timn/tests/nim/all/t10375c.nim(92, 19) fun2
/Users/timothee/git_clone/nim/timn/tests/nim/all/t10375c.nim(91, 26) fun1
/Users/timothee/git_clone/nim/Nim_prs/lib/system.nim(3024, 9) nimInternalNewException
/Users/timothee/git_clone/nim/timn/tests/nim/all/t10375c.nim(1, 1) template/generic instantiation from here
/Users/timothee/git_clone/nim/Nim_prs/lib/system.nim(3024, 9) Error: unhandled exception: cannot open: /tmp/nonexistant/bar.txt
VM callback stacktrace (compile nim with --stacktrace if needed)
/Users/timothee/git_clone/nim/Nim_prs/lib/system/io.nim(706) writeFile

 [IOError]
          raise newException(T, msg)
          ^

stacktrace inside VM callback

furthermore, when nim itself is compiled with --stacktace, the msg will also show the VM stacktrace (truncated to the useful part) eg:

stack trace: (most recent call last)
/Users/timothee/git_clone/nim/timn/tests/nim/all/t10375b.nims(18, 7)
/Users/timothee/git_clone/nim/timn/tests/nim/all/t10375b.nims(17, 20) fun2
/Users/timothee/git_clone/nim/timn/tests/nim/all/t10375b.nims(16, 7) fun
/Users/timothee/git_clone/nim/Nim_prs/lib/system/nimscript.nim(325, 16) cd
/Users/timothee/git_clone/nim/Nim_prs/lib/system.nim(3023, 9) nimInternalNewException
/Users/timothee/git_clone/nim/Nim_prs/lib/system.nim(3023, 9) Error: unhandled exception: No such file or directory
Additional info: "nonexistant"
VM callback stacktrace
/Users/timothee/git_clone/nim/Nim_prs/compiler/vm.nim(1231) rawExecute
/Users/timothee/git_clone/nim/Nim_prs/compiler/scriptconfig.nim(88) :anonymous
/Users/timothee/git_clone/nim/Nim_prs/lib/pure/os.nim(1367) setCurrentDir
/Users/timothee/git_clone/nim/Nim_prs/lib/pure/includes/oserr.nim(94) raiseOSError

 [OSError]
          raise newException(T, msg)
          ^

TODO for future PR's

• after this PR, it should be easy to catch C++ exceptions from CT FFI
• handle similarly opcNDynBindSym

links

• using readFile at compile time break exceptions - Nim forum

[krux02]

I can't find the test case.

[timotheecour]

done PTAL

[cooldome]

I have reviewed this PR and really like it. Only two comment I have:

• I don't think you need to introduce wInternalProc pragma, you can use compilerProc pragma and proc getCompilerProc*(g: ModuleGraph; name: string): PSym from magicsys instead.
• You have introduced argument mayRaise that use needs to be populated by the user. I would consider checkings {.raises[].} effect of symbol proc type instead and automatically infer if wrapped proc can raise exception or not.

[cooldome]

I have checked again and i think it safe to assume that wrapped proc can always raise exception, little overhead is added for potential exception handling.

[Araq]

The addition to system.nim is terrible and the feature itself is of questionable value. Yes, I know you will not stop until everything from os.nim is available in the VM, but I don't like it and the existing mechanism might suffice. But as others remarked the worst part is the new pragma and the system.nim additions neither of which are required afaict.

[stale[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. If you think it is still a valid PR, please rebase it on the latest devel; otherwise it will be closed. Thank you for your contributions.

[stale[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. If you think it is still a valid PR, please rebase it on the latest devel; otherwise it will be closed. Thank you for your contributions.