Background output prepending in REPL

[staticfloat]

It would be pretty cool to prepend output in the REPL when the main task is waiting for input. Use case: I spawn off a background task that is outputting stuff to STDOUT, and rather than having it spew gobbledygook all over the line where I am supposedly writing my next line of code, it gets placed above my current julia> prompt. Ideally, this would only happen when the julia> prompt was ready for input, if I have a computation running on the main task and it is printing out, it makes sense to not have this behavior.

This would (from a UI perspective) enable all sorts of neat things as useful background tasks, such as automatically precompiling packages after Pkg.build() in the background, or spawning off a long-running shell command, still interacting with Julia in the foreground, all while seeing the output of the shell command in a way that doesn't impact your current input.

[Keno]

Gallium does this for LLDB output.

[staticfloat]

Gallium does this for LLDB output.

Yes, precisely. Here it is in action, look at the Gallium.jl window on the right. @Keno, am I right in thinking that this file contains the pertinent logic from Gallium.jl for this?

[Keno]

No, it's here: https://github.com/Keno/Gallium.jl/blob/e840c2bafd4657c62b83a99bec5114442ebe385d/src/Gallium.jl#L104-L110

[staticfloat]

I hacked something up over the last few days, you can see it in all it's majesty, although it's not as seamless as I like due to us only having one thread available (despite running in a background task, it still freezes the CLI, especially on Pkg operations). It is, however, fun to play around with.

[vtjnash]

I've thought about using the following POC, which separates the two cursors (one for output, the other for input). It seems to work pretty well, but I can't figure out how to make commit_line look pretty due to the insane coupling in the LineEdit types and zero documentation.

diff --git a/base/LineEdit.jl b/base/LineEdit.jl
index 3169537..3d4595f 100644
--- a/base/LineEdit.jl
+++ b/base/LineEdit.jl
@@ -165,25 +165,15 @@ function complete_line(s::PromptState, repeats)
             show_completions(s, completions)
         end
     end
+    nothing
 end

-clear_input_area(terminal, s) = (_clear_input_area(terminal, s.ias); s.ias = InputAreaState(0, 0))
-clear_input_area(s) = clear_input_area(s.terminal, s)
-function _clear_input_area(terminal, state::InputAreaState)
-    # Go to the last line
-    if state.curs_row < state.num_rows
-        cmove_down(terminal, state.num_rows - state.curs_row)
-    end
-
-    # Clear lines one by one going up
-    for j = 2:state.num_rows
-        clear_line(terminal)
-        cmove_up(terminal)
-    end
-
-    # Clear top line
-    clear_line(terminal)
+function clear_input_area(terminal, s)
+    write(terminal, "\e[J") # erase from cursor to end of screen
+    s.ias = InputAreaState(0, 0)
+    nothing
 end
+clear_input_area(s) = clear_input_area(s.terminal, s)

 prompt_string(s::PromptState) = s.p.prompt
 prompt_string(s::AbstractString) = s
@@ -192,24 +182,22 @@ refresh_multi_line(s::ModeState) = refresh_multi_line(terminal(s), s)
 refresh_multi_line(termbuf::TerminalBuffer, s::ModeState) = refresh_multi_line(termbuf, terminal(s), s)
 refresh_multi_line(termbuf::TerminalBuffer, term, s::ModeState) = (@assert term == terminal(s); refresh_multi_line(termbuf,s))
 function refresh_multi_line(termbuf::TerminalBuffer, terminal::UnixTerminal, buf, state::InputAreaState, prompt = ""; indent = 0)
-    _clear_input_area(termbuf, state)
-
     cols = width(terminal)
     curs_row = -1 # relative to prompt (1-based)
     curs_pos = -1 # 1-based column position of the cursor
     cur_row = 0   # count of the number of rows
     buf_pos = position(buf)
     line_pos = buf_pos
-    # Write out the prompt string
-    write_prompt(termbuf, prompt)
-    prompt = prompt_string(prompt)
+    # the prompt string
     # Count the '\n' at the end of the line if the terminal emulator does (specific to DOS cmd prompt)
     miscountnl = @windows ? (isa(Terminals.pipe_reader(terminal), Base.TTY) && !Base.ispty(Terminals.pipe_reader(terminal))) : false
-    lindent = strwidth(prompt)
+    lindent = strwidth(prompt_string(prompt))

     # Now go through the buffer line by line
     seek(buf, 0)
     moreinput = true # add a blank line if there is a trailing newline on the last line
+    towrite = String[]
+    atcol = Int[]
     while moreinput
         l = readline(buf)
         moreinput = endswith(l, "\n")
@@ -217,8 +205,8 @@ function refresh_multi_line(termbuf::TerminalBuffer, terminal::UnixTerminal, buf
         llength = strwidth(l)
         slength = sizeof(l)
         cur_row += 1
-        cmove_col(termbuf, lindent + 1)
-        write(termbuf, l)
+        push!(atcol, cur_row == 1 ? 0 : lindent + 1)
+        push!(towrite, l)
         # We expect to be line after the last valid output line (due to
         # the '\n' at the end of the previous line)
         if curs_row == -1
@@ -234,7 +222,8 @@ function refresh_multi_line(termbuf::TerminalBuffer, terminal::UnixTerminal, buf
                 if curs_pos == cols
                     # only emit the newline if the cursor is at the end of the line we're writing
                     if line_pos == 0
-                        write(termbuf, "\n")
+                        push!(atcol, 0)
+                        push!(towrite, "\n")
                         cur_row += 1
                     end
                     curs_row += 1
@@ -247,16 +236,41 @@ function refresh_multi_line(termbuf::TerminalBuffer, terminal::UnixTerminal, buf
         lindent = indent
     end
     seek(buf, buf_pos)
+    cursor = eof(buf) ? ' ' : read(buf, Char)
+    seek(buf, buf_pos)

-    # Let's move the cursor to the right position
-    # The line first
+    # do the output
+    write(termbuf, "\e[J") # erase from cursor to end of screen
+
+    # Reserve the output rows (without affecting col) but dealing with scrolling
+    for j = 1:cur_row
+        write(termbuf, "\eD") # move down N
+    end
+    for j = 2:cur_row
+        write(termbuf, "\eM") # move back up N-1
+    end
+    write(termbuf, "\e7\e[1G") # record resulting cursor position and move to column 0
+
+    write_prompt(termbuf, prompt)
+    for i = 1:length(towrite)
+        atcol[i] > 1 && cmove_col(termbuf, atcol[i])
+        write(termbuf, towrite[i])
+    end
+
+    # Now work on drawing the cursor
+    # move to the right line
     n = cur_row - curs_row
-    if n > 0
-        cmove_up(termbuf, n)
+    for j = 1:n
+        write(termbuf, "\eM") # move back up N
     end

-    #columns are 1 based
+    # and the right column
+    # columns are 1-based
     cmove_col(termbuf, curs_pos + 1)
+    write(termbuf, "\e[7m") # invert color of cursor
+    write(termbuf, cursor)
+    write(termbuf, "\e8\eM") # write out virtual cursor and restore cursor to known point
+                    # (this is quite suboptimal if the terminal is not tall enough to display the whole input, but what can we do?)

     # Updated cur_row,curs_row
     return InputAreaState(cur_row, curs_row)
@@ -449,7 +463,7 @@ end
 function edit_insert(s::PromptState, c)
     str = string(c)
     edit_insert(s.input_buffer, str)
-    if !('\n' in str) && eof(s.input_buffer) &&
+    if false && !('\n' in str) && eof(s.input_buffer) &&
         ((position(s.input_buffer) + sizeof(s.p.prompt) + sizeof(str) - 1) < width(terminal(s)))
         # Avoid full update when appending characters to the end
         # and an update of curs_row isn't necessary (conservatively estimated)
@@ -1288,9 +1302,10 @@ end
 function commit_line(s)
     move_input_end(s)
     refresh_line(s)
+    cmove_down(terminal(s), state(s, mode(s)).ias.num_rows)
     println(terminal(s))
-    add_history(s)
     state(s, mode(s)).ias = InputAreaState(0, 0)
+    add_history(s)
 end

 """
@@ -1548,7 +1563,7 @@ end

 run_interface(::Prompt) = nothing

-init_state(terminal, prompt::Prompt) = PromptState(terminal, prompt, IOBuffer(), InputAreaState(1, 1), #=indent(spaces)=#strwidth(prompt.prompt))
+init_state(terminal, prompt::Prompt) = PromptState(terminal, prompt, IOBuffer(), InputAreaState(0, 0), #=indent(spaces)=#strwidth(prompt.prompt))

 function init_state(terminal, m::ModalInterface)
     s = MIState(m, m.modes[1], false, Dict{Any,Any}())

[staticfloat]

I agree that the approach in that package (and the modified Julia necessary) isn't a good one; using only one cursor is rife with problems, particularly when you have other tasks that can write to STDOUT at any time, the user is typing in whenever they wish, etc... What we really need is some way to have multiple output streams that are completely independent of eachother, and make writing to those streams nice and atomic.

[vtjnash]

yeah, that's effectively what the above diff does (within the limitations of going through a terminal emulator). I just gave up trying to figure out how to delete the extra cursor after commit_line, so there's a bit of linenoise left around afterwards and hoped someone else might want to kick it around some.

[davidanthoff]

This is exactly what we need to make progress with the REPL in the VS Code julia extension.

I won't even try to tackle this, if @vtjnash couldn't figure out how to do this. Is @Keno the one who could make progress on this? Also, it is not clear whether there is code in Gallium already that does this?

Maybe this could get a new priority, given that we have a very clear use-case for this now, the REPL in VS Code.

[ihnorton]

@davidanthoff I hacked something up that might work for you, based on what I understand of your requirements: https://gist.github.com/ihnorton/ee3ef6e2858fdc567681113b81838af2

However, it requires a small patch to base, to get access to the prompt state, and I'm not sure if this is something we want to expose. Posting here to ask for suggestions of another way to get this handle:

diff --git a/base/LineEdit.jl b/base/LineEdit.jl
index 704dc14..36f2a7d 100644
--- a/base/LineEdit.jl
+++ b/base/LineEdit.jl
@@ -1566,9 +1566,10 @@ function init_state(terminal, m::ModalInterface)
     end
     s
 end
-
+global _state
 function run_interface(terminal, m::ModalInterface)
     s::MIState = init_state(terminal, m)
+    global _state = s
     while !s.aborted
         p = s.current_mode
         buf, ok, suspend = prompt!(terminal, m, s)

[davidanthoff]

Bump, any chance that this patch that @ihnorton suggested could make it into Base? I assume @Keno is the person to look at that? @KristofferC also suggested that this is the way to solve a problem in our VS Code extension.

[staticfloat]

@davidanthoff I just opened a PR with something really similar to Isaiah's patch in #24735

[davidanthoff]

Cool! Then I just need to understand the REPL stuff and we might actually be able to integrate it better in VS Code ;)

[petvana]

This is precisely the feature I was looking for. Has anyone found a satisfactory solution over the recent years? Or, do you know about any package that actively uses prepending?

(I've found that the async printing did not make it to https://github.com/JunoLab/atom-julia-client/pull/394 before merging, and there is no other reference to #24809.)