Add better tools for calling CLI commands in editor

[NovaDC]

[This proposal has been edited from it's orignal version, use the post history if you wish to view the older versions]

Describe the project you are working on

In general, projects that use cli commands in the editor (including the godot editor itself, such as when using rcedit.exe when exporting for windows).

Describe the problem or limitation you are having in your project

When making editor tools, OS.execute and OS.execute_with_pipe is sometimes just not enough.

When executing something that can take a while, the OS might assume the editor freezes (even though its just blocked by another process).

Even worse, the user might also presume this! This can cause issues when - say - modifying or generating a large file, as this can possibly stop the process half way through.

Also, though not as big a problem, common commands bound to a name on a system (such as python => bin/python/python3) can be used as there is no easy way to locate these, in a cross platform fashion.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

It would be really nice to have the commands run, regardless of platform, to have a concise way to chow stdin and out to the user of the editor. Tasks such as linting, formatting, heck - some tools called in the builtin export process would all be much easier to understand if there was some way for the user to see the output of these commands as they happen and view the progress they print out as text. This can also help aid the sense of the editor being frozen when a process naturally is taking alot of time for otherwise valid reasons, and can aid in debugging these commands as well.

Also, these special commands should be put in EditorInterface, instead of in the OS singleton. Putting this functionality in the EditorInterface makes it both implicitly clear that these functions are made specifically in editor use, and for possible future sandboxing of commands begin run via OS.execute (if a future platform requires this). Further more this also ensures that a function like this isn't called from a non-editor instance by accident.

Also, having this functionality being extended to a execute_with_pipe version as well allows for potential interactive system cli in the editor, if necessary.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

I have already tried to implement this in a common addon on my own (though, its not entirely the best way to implement this, and as this called the system's terminal to display the output instead. I have not given it as through a look over as I should of to simply submit a c++ version of this in a pull request). See here.

The change I specifically want to see is to add (both in the underlying c interface and in gdscript) the following non-static methods to EditorInterface:

• execute(command:String, args:PackedStringArray, read_stderr:bool = true, read_stdin:bool = false, keep_open:bool = false) -> int
• execute_with_pipe(command:String, args:PackedStringArray, read_stderr:bool = true, read_stdin:bool = false,, keep_console_open:bool = false) -> Dictionary
• execute_new_instance(args:PackedStringArray, read_stderr:bool = true, read_stdin:bool = false,) -> int (basically create_instance but using EditorInterface.execute instead)

It's also with not that these methods should "block" the function their in when called, but not the editor itself. The loading ui should be called (greying out the editor and blocking input, like when exporting or loading) and the application should spool in a way that assures the OS doesn't think it's frozen.

There also must be a interface made for stdin/out interaction with these commands (a terminal like interface, perhaps a text output as a rich text box on the top and a text entry line below it for interaction). The stdin section can be hidden for read only interaction, if called with the right argument.

It would also be nice to have a option for changing the cwd of the command line before executing, but this is another thing that confuses me on how to implement, and might provide diminishing returns on the real value of this feature. For this reason I have decided to leave out the additional optional String type parameter of cwd in all the methods above.

It would also be nice (though not entirely mandatory) to have a function like command_path(common_name:String, path:String = "") -> String be added to EditorInterface. This command has a similar purpose to something like python's shutil.which, with cmd being common_name and path being path. A fucntion like this enabled tools using the editor specific execute functions to also access the common tools of the OS, allowing for easier self configuration of some tools without requiring full command line functionality to be replicated in godot itself for every platform that has a editor and an accessible command line.

If this enhancement will not be used often, can it be worked around with a few lines of script?

Now? Yes. If this is not implemented, the a solution (well, perhaps a slightly better built version of the existing solution I gave above, stated above) is already doable. But that's not the point of this request. The point is to make a explicit divide between the executing of a external program in editor and in game (non-editor) separate, and to provide several quality of life features that not only enrich the usage of external programs for addons but the editor itself.

There is also a point to be made that this functionality would be used alot if implemented. This would be a better choice to use when calling tools already used when exporting (for example, rcedit.exe when exporting for windows).

Is there a reason why this should be core and not an add-on in the asset library?

See above.

[NovaDC]

I am actively interested in making the a pull request, however I would like to first see if I am overlooking things or if there might be a better way to do this.

[NovaDC]

Also, the methods stated above may have the term shell_ or cli_ prepended to their names in order to differentiate them form their OS method counterparts.

[Calinou]

You can't execute shell specific commands (ex. echo "hello" => File 'echo' not found. when using OS.execute).

You can if you run the shell directly (e.g. /bin/bash on Linux), but relying on shells makes your code more OS-specific. Additionally, it's a bit slower than running the program directly, which can be important if you're using OS.execute() hundreds of times or more.

[NovaDC]

That solution is exactly what I am doing at the moment. The more important part is that the user sees the proper terminal output and can do proper terminal input to the command being run.

Perhaps this specifically being in a separate window and application isn't wholly necessary, but doing this is less work than building a whole ui for stdin/out of a command being run specifically for the editor.

[KoBeWi]

You can use execute_with_pipe() to get live output.

[NovaDC]

You can use execute_with_pipe() to get live output.

Perhaps this specifically being in a separate window and application isn't wholly necessary, but doing this is less work than building a whole ui for stdin/out of a command being run specifically for the editor.

execute_with_pipe() also doesn't provide the a way to grey out the editor when blocking (or for shell specific commands to be used though I can see how shell specific commands aren't a strict necessity, I would personally prefer these)

[KoBeWi]

You can gray out the editor manually by either using modulate or putting ColorRect over everything.

[NovaDC]

I suppose the alternative implementation would be to use execute_with_pipe() internally and the call the loading ui to grey out the editor and block input except for the additional ui being made that would act as a stdin/out interface for the application, but applications would still need to be resolved somehow (ex. python ->bin/python/python3) (I believe this can be done by reading the path environment variable, not familiar with how this specific resolution is done), but that is additional work to provide a feature already present on a system that can run an editor.

This is however another option for implementing these features.

I can also see doing it this way as beneficial as this could provide a way to interact with a terminal instance in the editor in the furtue, if a stdin/out ui exists in the editor (just throw it into the bottom dock and tell it to run cmd.exe or whatever works for the OS). Also this does remove the overhead of running multiple separate shells as Calinou mentioned.

[bruvzg]

You can't execute shell specific commands (ex. echo "hello" => File 'echo' not found. when using OS.execute).

You can, but you need to call a specific shell, e.g cmd /c on Windows.

When executing something that can take a while, the OS might assume the editor freezes (even though its just blocked by another process).

That's what the blocking argument in the execute_with_pipe is, set it to false and it will return immediately, you can read and display output from the returned pipe handle as you want.

Also, no matter the terminal of choice (as set in the "filesystem/external_programs/terminal_emulator" editor setting), these commands will always run outside the terminal!

This setting has nothing to do with running commands.

Also, the open_console argument is Windows only and does nothing on other platforms.

[NovaDC]

I might not have explained this correctly. I understand that you can run the shell's executable by using its executable path in OS.execute, as this is what I was already doing in my (kinda funky) example-prototype-thing linked in the original post. I understand that that's how shells work, I just want this functionality to be baked into a function that already knows the proper way to call a platform appropriate shell (appropriate flags, proper encapsulation of the arguments, etc).

These commands, when run like this, must block the editor. It should block it in such a way that it makes it clear to the user that a command is being run and not to touch anything while it is running (preferably blocking out editor input entirely), without the OS thinking that the editor is frozen. These commands must show their outputs, as the results of these commands report exactly what it's doing, useful for debugging and monitoring what stage the task is in.

I can see that calling shells are not entirely a practical choice, however, but this would result in more work specifically, to substitute the native shell you have to implement the features as stated before and also make a new stdin/out ui and a way to resolve command names (python => bin/python/pyhton3).

[NovaDC]

After having a nap it appears I was unreasonably stuck on the concept of this usign the builtin terminal of the os. After rereading the comments here I can see now how making a separate ui would be a substantially better choice. I'll change the proposal to match this.

I'm also torn over weather or not there has to be a way to resolve the application paths of common commands (python => bin/python/pyhton3). To implement this, there isn't a cross platform method. On unix systems, the which command seems like the best bet, with windows having the where command; but that brings up an issue of finding a way to call the command that finds other commands.

[NovaDC]

I will attempt to release a gdscript addon that encapsulates calls to the systems native terminal instead.

I currently lack the knowledge of stdio interfaces to a degree where I would be comfortable code, as there are may edge cases and possible security risks that I might not know of. I also lack a OSX platform to tests my code on, and a feature like this should ideally be something that can work on any desktop platform. I'm also not the best at making UIs though I can see UIs being improved when making a pull request.

I will update this proposal if I find myself able to implement this into the Godot core, or close this if this proposal in implemented in part or in whole in a future update. However, I still find that a feature like this will be very helpful to use in the future, and wish for it to be added to some extent to the Godot API.