Ability to print and log script backtraces

[reduz]

• Changes how stack information is stored in GDScript to a reverse linked list.
• This makes it fast enough to leave it enabled all time time on debug.
• Added a new script function script_backtrace() to get a string with the current script backtrace.
• Added the script backtrace to the logger. Errors now include the full currently running script backtraces.
• Added script backtrace printing to console logs (Windows, MacOS, Linux) using blue color.
• Additionally, added a custom Logger class to allow users intercept the internal messages/warnings/error (supersedes #87576).

How it looks:

using script_backtrace()

Results in the following output:

Logger

Backtraces are now added to the loggers. Here is how it looks for ANSI/Windows console loggers:

[realcoloride]

Extremely wanted feature that could make it way easier to debug. Love this

[bruvzg]

Not sure how reliable it will be, but it might be worth calling it from the crash handler as well (tested on macOS only and seems to be working):

diff --git a/platform/macos/crash_handler_macos.mm b/platform/macos/crash_handler_macos.mm
index c370422bfa..91f76dc3b5 100644
--- a/platform/macos/crash_handler_macos.mm
+++ b/platform/macos/crash_handler_macos.mm
@@ -31,6 +31,7 @@
 #include "crash_handler_macos.h"
 
 #include "core/config/project_settings.h"
+#include "core/object/script_language.h"
 #include "core/os/os.h"
 #include "core/string/print_string.h"
 #include "core/version.h"
@@ -171,6 +172,9 @@ static void handle_crash(int sig) {
 	}
 	print_error("-- END OF BACKTRACE --");
 	print_error("================================================================");
+	print_error(ScriptServer::get_current_script_backtrace());
+	print_error("-- END OF SCRIPT BACKTRACE --");
+	print_error("================================================================");
 
 	// Abort to pass the error to the OS
 	abort();

[RedMser]

How does the new function relate to the existing @GDScript.get_stack() and print_stack()? I assume there are technical differences, but these are not really made clear enough to the average user (when should I use this new function vs the existing ones?).

[farfalk]

In addition to what RedMser asked, how does this relate to https://github.com/godotengine/godot/pull/87576 ?

[reduz]

@RedMser

This new one works always, even if not running the debugger and it also is generic for all languages. print_stack() was created mostly for testing and should be deprecated and removed since its pretty useless.

@farfalk

Both PRs should work great together

[permelin]

Did a quick test script with a function that is recursively called 500 times inside a for loop. To my surprise, this branch is 5% faster than master.

(Pushing the call stack beyond a depth of 506 or 507 will SIGSEGV, but that happens in master too.)

[dalexeev]

Can script_backtrace() return data in a structured form (Array[Dictionary]), like get_stack()? It also looks like this PR will resolve godotengine/godot-proposals#105.

[reduz]

@dalexeev I honestly don't see a lot of use for this, since the main use case is to print it, but I think adding another function script_stack() may be good for this, that could also include more stack information if you want.

[Mickeon]

A bit out of the loop, does https://github.com/godotengine/godot/pull/87576 essentially supersede this PR?

[reduz]

@Mickeon No, the other way around.

[radiantgurl]

Is the SIGSEGV gonna be fixed so i could try it out with #94364 ?

[kleonc]

Can script_backtrace() return data in a structured form (Array[Dictionary]), like get_stack()? It also looks like this PR will resolve godotengine/godot-proposals#105.

@dalexeev I honestly don't see a lot of use for this, since the main use case is to print it, but I think adding another function script_stack() may be good for this, that could also include more stack information if you want.

Maybe the main use case is indeed to print it the default way, but for sure it needs to allow customization. One might want to print it but not with the default formatting (users do use custom logging for nice/clear output up to their liking). E.g. one could make the outputted stack trace entries clickable within the Godot editor, using BBCode (so they would take you to the relevant code).

With backtrace provided as a string one would need to do some reverse engineer parsing to get the actual data from it, which is not user friendly and not reliable.

I would argue that what's fundamental here is providing the actual stack data (e.g. as Array[Dictionary]). An easy way to obtain the whole backtrace string (with an arbitrary default formatting) I would consider as a nice-to-have bonus on top of that.

So I'm suggesting that the base should be:

• Logger._log_error() instead of taking a preformatted string script_backtrace argument, it should take script_stack argument being the actual stack data.
• Global script_stack() function returning the actual data, not a string. Could be named differently of course.

Then I wouldn't really mind what would be the additional stuff on top of that:

• Some kind of script_stack_to_string(Array[Dictionary]) utility function for default formatting?
• Additional script_backtrace() function? (same as script_stack_to_string(script_stack()))
• Etc.

---

This kinda remainds me of Node.print_tree{_pretty}. I'm guessing these methods were initially added as "the main use case". Later users requested/added Node.get_tree_string{_pretty} as it makes sense to output same thing to some UI or file. Note that these methods still do not handle cases when you want custom formatting, or to print custom data for each Node etc. But that's not a big problem, because Node tree data is available and thus everyone can manually traverse it outputting whatever/however they like.

In case of a script stack trace this would be a problem, because the actual stack trace data would be not obtainable in the first place.

[felipejfc]

Either this or https://github.com/godotengine/godot/pull/87576 need to be merged with high priority. Not being able to log unhandled exceptions to a remote system is a blocker to launch any serious mobile game with Godot. For the SIGSEV that's happening here, could you fix it or give directions @reduz ?

[mihe]

I was asked by @reduz to try to take this across the finish line a while back, and was underway with diagnosing the SIGSEV at one point, but other priorities have kept getting in the way.

If I remember correctly the SIGSEV in the unit tests are largely the same issue I mentioned above with regards to stale pointers as a result of await, but I didn't have time to actually experiment with any fixes for it.

@reduz also expressed an interest in seeing some variation of #93648 merged into this PR, since you would then get not only the script backtrace as part of log_error, but also the native backtrace.

How that native backtrace holds up across the various platforms remains to be seen, but seeing as how symbolication will likely be impossible on mobile platforms I suspect the better option might be to have the backtraces be returned in some kind of structured format, rather than just a string, similar to what has been suggested in this thread by @kleonc already. That way you could at least get the stack frame pointers as part of the structured format and send those off to some service somewhere to be symbolicated, if you wanted to, or just pass the structured format into some function to be symbolicated locally (assuming you actually have the symbols and the means to do so) and then finally into some other function to be string-formatted.

Anyway, I'll try to get to this in a (hopefully) near future, if no one else beats me to it.

[juse4pro]

Glad you are taking care of this topic @mihe 😃 This issue is a huge show stopper for our online game since Godot 4 came out. Unhandled C# exceptions that don't crash the game are terrible to debug. So this with PR we could gracefully get our logs / Sentry report and be happy.

Compared to Godot 3 a missing "feature". 😄

Is there any estimation if this PR could be merged into 4.4?

[felipejfc]

Possibly into 4.3 given it's a very small change that's easy to backport and is super important to any production project

[AThousandShips]

This is an enhancement so it's not going to go into 4.3 unless it's considered really exceptional, we don't cherry-pick enhancements