Feature request: add better debugging facilities (breakpoints, step-by-step execution, etc.)

[fingolfin]

This is inspired by this old feature request: https://bitbucket.org/gap-system/gap/issues/26/debug-mode

What I'd envision is to improve the break loop, which is already nice, but not quite there. Things I'd like to have:

1. Setting and clearing break points dynamically, during runtime. Perhaps there could be a command SetBreakpoint(func[, line]); which sets a breakpoint at the start of a function, and optionally, at a specific line. Of course there should also be a ClearBreakpoint, and also ClearAllBreakpoints(func) (and ideally also a global ClearAllBreakpoints(), though whether that is doable would depend how breakpoints are implemented).

• Displaying a function should also mark any breakpoints.
• Bonus points for conditional breakpoints, and being able to temporarily disable them

2. A way to step through code. That is, if one is in a break loop due to a break point, one should be able to enter a command StepNext() and it would execute the next "line" of code. Or perhaps the next statement. (Technically, that could work by inserting a temporary breakpoint after the next statement).

3. A similar StepInto() would also be nice, which would not step over the next line (statement), but, if it is a function, into it. However, I am not so sure how exactly that should work in GAP, in lights of method dispatch... Perhaps one should have two (or more) variants of it; one which skips over the method dispatch, and would step into whichever method got selected; and another which would step into the method dispatcher, and would be useful to debug that; but implementing that would likely be much, much harder, due to the fact that the debugger itself likely would make use of method dispatch, and / or anything the user enters might...

4. For power users, a debugger should of course also allow stepping via hotkeys... In many IDEs, function keys can be used to step to the next instruction. Would be nice to have here, too.

5. Better state introspection in the break loop: Better backtrace output; additional functions to display the locals at the current / each level; .... probably more I am not thinking of right now

6. more... ? Additional ideas are welcome :-).

Note that I have done some work regarding improved backtraces at least, and also on fixing DownEnv/UpEnv. See also issue #358

As to technical details: I must admit I am not so familar with the relevant kernel code, and have not at all looked into how we might be able to do this, exactly. But my hope is that perhaps we can steal a bit somewhere, to mark "instructions" (statements) in function body as having an active breakpoint. Isn't the new profiling code doing something along these lines?

Then setting / clearing breakpoints would be relatively easy, and it should be possible to implement this with little to no speed overhead. For conditional breakpoints, I'd imagine all the extra data could be stored in a separate slot in the function body... ?!

[ChrisJefferson]

We could probably find a spare bit in instructions, but the problem would be figuring out how to set it -- we don't currently really have a way of walking through a function object, looking for statements on particular lines. One would have to look at this carefully.

I can imagine how to do breakpoints with cost (no cost when they are not being used) by hooking into the code used by line-by-line profiling. As it considers every statement, it could just check if we are now in a statement where we want to break. At some point I had a dodgy hacked up version of this. This would have the advantage of allowing lots of nice things, we can just abstract the profiler to allow a (C-level) function to be called every time a function is entered/exited, and a new line is executed.

[ChrisJefferson]

I should add, I won't have time to work on this in the near future, but if anyone else wants to, I can quickly explain the code you would want to look at. Also, it would be nice to do a general cleanup of the profiling code at the same time (in particular, there should be a mode which doesn't generate a massive JSON file, but instead builds a more compact data structure directly which scales better)

[fingolfin]

Note that there already seems to be some undocumented debugging facilities: See lib/debug.g, etc/debugvim.txt, etc/debug.vim.