On Windows 10, at launch of LLDB, program is unable to access stdout or stdin

[gandalfas]

Issue Description

On Windows 10, at the launch of Swift's custom LLDB by the Swift extension for VS Code (vscode-swift), the user program's output from the 'print()' statement is nowhere to be seen, and the attempt to do 'readline()' gets an EOF result.

The issue seems most likely to be in the Swift extension for VS Code, or possibly in the CodeLLDB extension. This is based on the 'Variations' section, below.

This is in the global settings.json file, which correctly points to the Swift custom LLDB:

{
    "lldb.library": "C:\\Library\\Developer\\Toolchains\\unknown-Asserts-development.xctoolchain\\usr\\bin\\liblldb.dll"
}

Discovered during work on #301.

Let me know if you need me to provide any additional information.

Environment

OS: Windows 10 VSCode version: 1.67.1 Swift extension version: 0.5.2 CodeLLDB version: 1.7.0 Compiler: Swift 5/04/22 'latest snapshot of main': swift-DEVELOPMENT-SNAPSHOT-2022-05-04-a-windows10.exe

• Note: The results below differ substantially if the slightly earlier release, Swift 5.6.1, is used - see #301.

Test Case

md hello cd hello swift package init --type=executable place these lines into main.swift:

import Foundation

print("Hello, world!")
print("enter something  ->", terminator: "")

guard let answer = readLine() else {
    print("End of File was reached - aborting...")
    exit(1)
}
print("readline() did not see EOF")

open folder with VS Code compile with: Terminal > Build Debug hello hello.exe is built. set a breakpoint on 'Hello,world' line. click Run > Start Debugging, and then step through the program.

Expected Behavior

This expectation is based on one of the 'Variations' - see below. At this point you should see the following in VS Code's lower pane > Terminal tab. When the prompt "enter something" appears, you should be able to enter text, for example "abc". Control should skip over the 'else' to the print statement at the bottom of the program. At the end of execution, the Terminal tab should show:

Hello, world!
enter something  ->abc
readline() did not see EOF

Actual Behavior

A new command prompt window is created, titled with 'hello.exe'. It remains empty throughout program execution.

The output of the print statements is nowhere to be seen - neither in the VS Code lower pane in tab 'Terminal', nor in the new empty command prompt window.

As you step over the 'readline()' statement, the program fails to wait for you to enter text, control simply falls into the 'else' part of the guard, where EOF is handled.

This output appears in the lower pane, tab 'Output', channel LLDB:

configuration: {
  type: 'lldb',
  request: 'launch',
  name: 'Debug hello',
  program: '${workspaceFolder:hello}/.build/debug/hello',
  args: [],
  cwd: '${workspaceFolder:hello}',
  preLaunchTask: 'swift: Build Debug hello',
  __configurationTarget: 5,
  relativePathBase: 'k:\\Kim Shared\\CodeLab\\SwiftTests\\hello'
}
[adapter\src\terminal.rs:104] FreeConsole() = 1
[adapter\src\terminal.rs:105] AttachConsole(pid) = 1
[adapter\src\terminal.rs:109] FreeConsole() = 1
[2022-05-17T23:49:11.082Z ERROR codelldb::debug_session] Internal debugger error: Invalid frame reference: 1001

There seems to be some variability in this error log - in another similar case, shown in #301, in the comments by me starting with "Alrighty", I had this instead:

[adapter\src\terminal.rs:104] FreeConsole() = 1
[adapter\src\terminal.rs:105] AttachConsole(pid) = 1
[adapter\src\terminal.rs:109] FreeConsole() = 1
ERROR(Python) 20:38:54 codelldb: Traceback (most recent call last):
  File "c:\Users/Kim/.vscode/extensions/vadimcn.vscode-lldb-1.7.0/adapter\codelldb.py", line 152, in evaluate
    value = evaluate_in_context(pycode, is_simple_expr, context)
  File "c:\Users/Kim/.vscode/extensions/vadimcn.vscode-lldb-1.7.0/adapter\codelldb.py", line 288, in evaluate_in_context
    return eval(code, eval_globals, eval_locals)
  File "<input>", line 1, in <module>
  File "c:\Users/Kim/.vscode/extensions/vadimcn.vscode-lldb-1.7.0/adapter\codelldb.py", line 269, in __missing__
    raise VariableNotFound(name)
codelldb.VariableNotFound: Variable 'vars' not found

[2022-05-12T01:38:54.750Z ERROR codelldb::debug_session] Variable 'vars' not found
Debug adapter exit code=0, signal=null.

Variations

When I comment out the "lldb.library" entry in settings.json, thus forcing the standard LLDB (poorly named, I meant the LLDB packaged with the CodeLLDB extension) to be used instead of the Swift custom LLDB, then do Run > Start Debugging, the print() statement output appears in the lower pane of VS Code, in the Terminal tab; likewise, for readline(), the user enters text in the same Terminal tab. That feels like the correct behavior to me, for VSCode-based debugging. This is the basis of my expected behavior, above.

When I run the Swift LLDB from the command prompt, the program uses stdin/stdout just fine, and that occurs in a command prompt window that lldb launches for the program.

Likewise when I run the program directly from the command prompt (no LLDB), the prog uses stdin/stdout just fine in that same command prompt window.

Taking all these variations together, these indicate that the source of the problem must be in the setup done in the Swift extension as it prepares to invoke the LLDB debug adapter, or possibly the debug adapter is mishandling something when the lldb.library is set. If no objections, we propose that we begin the investigation in the Swift extension (see #301 at the bottom).

[gandalfas]

argg, @adam-fowler , I just saw your comment of yesterday. I did check for any follow-up from you or @compnerd on #301 just before creating this issue 309, but alas, I was looking at a stale view.

So, for good or bad, I've already created this issue, so I am going to paste your reply here, and then let's continue any discussion of this issue here, in 309.

@compnerd I've had a quick look at this and sorry to pass the buck but I think this is more a Swift lldb issue. Running from the command line the standard lldb prints directly to the command line any output from the application. If you use the swift lldb it opens up a new window for the output. I'm assuming this difference is what is causing the issue inside vscode-swift as there is no output to stdout/stderr it is all redirected to the window that opens.

[compnerd]

Hmm, I don't see how lldb would create a new console for the application, it does a standard process creation AFAIK. What is standard lldb here for context? Is it the lldb from llvm.org? Built at what revision?

[gandalfas]

I am trying to reproduce Adam's case:

the standard lldb prints directly to the command line any output from the application

but I get different results. when I put the standard lldb foremost in my PATH, I get:

--> lldb hello

Fatal Python error: init_sys_streams: can't initialize sys standard streams
Python runtime state: core initialized
Traceback (most recent call last):
  File "C:\Program Files\Python\Python310\Lib\io.py", line 54, in <module>
ImportError: cannot import name 'text_encoding' from 'io' (unknown location)

my standard lldb is installed in:

c:\Users\Kim\.vscode\extensions\vadimcn.vscode-lldb-1.7.0\lldb\bin

and that is the same lldb.exe that Run > Start Debugging calls (per Process Explorer). This was installed by the LLDB extension for VS Code. That lldb\bin dir also contains its own copies of these python files: python3.dll and python39.dll. These are pulling io.py from my python 3.10 install (even when I remove that from the PATH). I won't be able to shed any light on standard lldb's behavior on my machine unless I can get it to ignore python310 folders.

[gandalfas]

also, Adam said

If you use the swift lldb it opens up a new window for the output. I'm assuming this difference is what is causing the issue inside vscode-swift as there is no output to stdout/stderr it is all redirected to the window that opens.

but that differs from what I see - no output is directed to the newly opened window.

[gandalfas]

A thought: since commenting out the "lldb.library" entry in settings.json makes things work as expected (stdout and stdin are in VS Code's 'Terminal' pane), could we assume that whoever is processing that setting is messing up the redirecting of stdout and stdin to the Terminal pane? And would that be some component of the LLDB extension?

[adam-fowler]

What is standard lldb here for context? Is it the lldb from llvm.org? Built at what revision?

When I say standard LLDB I mean the one that is packaged with vscode-lldb. Looking at the commit comments from vscode-lldb this would be from https://github.com/llvm/llvm-project, build id 1927 (not totally sure what build id indicates though).

but that differs from what I see - no output is directed to the newly opened window.

Sorry you are right when run via vscode-lldb no output is display to the window.

but I get different results. when I put the standard lldb foremost in my PATH

You need to resolve your python issues before you can compare the results.

As I said above the two versions of lldb when run from the command line react differently. We need to resolve this before looking at vscode-swift/lldb.

[compnerd]

Thinking about this a bit: the new window might be the result of AllocConsole. There is nothing that can be done about that - it is how console applications work on windows. https://docs.microsoft.com/en-us/windows/console/allocconsole

I think that we should look at how the inferior is launched. I suspect that this might be a bug in lldb-server. One quick test would be to try a “remote” (it can be run on the same host) debug session with ds2 (https://github.com/compnerd/ds2).

[gandalfas]

the new window might be the result of AllocConsole

yes, agreed, and I believe that is a valid technique to achieve clarity, i.e., when lldb is run as a CL tool, it would want the debugger commands and cmd outputs to be segregated from the program's interactions with the user, so 2 consoles is a good way to accomplish that.

You need to resolve your python issues before you can compare the results.

yes, agreed. I had updated my notes, perhaps after you read the initial notes, to indicate the same. I have a couple ideas and will report back.

[gandalfas]

I was able to get the standard LLDB to ignore my python310 folders, by doing:

rem PYTHONHOME=C:\Program Files\Python\Python310
rem PYTHONPATH=C:\Program Files\Python\Python310\Lib;C:\Program Files\Python\Python310\Lib\site-packages;
set PYTHONHOME=c:\Users\Kim\.vscode\extensions\vadimcn.vscode-lldb-1.7.0
set PYTHONPATH=c:\Users\Kim\.vscode\extensions\vadimcn.vscode-lldb-1.7.0\Lib;c:\Users\Kim\.vscode\extensions\vadimcn.vscode-lldb-1.7.0\Lib\site-packages;

This is in addition to the previous change to add the standard lldb foremost in my PATH, so that the one under C:\Library is not seen. With all that, the standard LLDB had the identical behavior described by Adam:

Running from the command line the standard lldb prints directly to the command line any output from the application.

I.e., the program output and input are intermingled with the LLDB command output and input:

    frame #0: 0x00007ff6146c127c hello.exe`main at main.swift:10:7
   7    
   8    
   9   
-> 10   print("Hello, world!")
   11
   12
   13   
(lldb) n
(lldb) Hello, world!                                                          ####### output from program
Process 7348 stopped                                                       ####### output from LLDB
* thread #1, stop reason = step over
    frame #0: 0x00007ff6146c1364 hello.exe`main at main.swift:15:7
   12
   13  
   14
-> 15   print("enter something  ->", terminator: "")
   16
   17   guard let answer = readLineaa() else {
   18       // EOF should NOT happen from std input.  sth is terribly wrong.
(lldb) n
enter something  ->Process 7348 stopped                      ####### output from program;  output from LLDB
* thread #1, stop reason = step over
    frame #0: 0x00007ff6146c146e hello.exe`main at main.swift:17:28
   14
   15   print("enter something  ->", terminator: "")
   16
-> 17   guard let answer = readLineaa() else {
   18       // EOF should NOT happen from std input.  sth is terribly wrong.
   19       print("\n\n\nEnd of File was reached - aborting...")
   20       exit(1)
(lldb) n                                                                          ####### user input to LLDB
(lldb) aaabbbccc                                                              ####### user input to program
Process 7348 stopped
* thread #1, stop reason = step over
    frame #0: 0x00007ff6146c15b8 hello.exe`main at main.swift:23:7
   20       exit(1)
   21   }
   22
-> 23   print("readline() did not see EOF")
   24
   25
   26   /*
(lldb) n                                                                    ####### output from LLDB
readline() did not see EOF                                                ####### output from program
Process 7348 stopped                                                  ####### output from LLDB 

So, clearly the Swift custom LLDB was changed to enable 2 separate consoles, one for LLDB and one for the program. I hope this helps.

Possibly that may have to be undone in order to get Swift custom LLDB working under VS Code. However, my own 2 cents would be that the optimal solution would be to fix the stdout/stdin redirecting while retaining the 2 separate consoles. Otherwise we are left with choosing between (A) cmd line LLDB working great with 2 consoles and VS Code-based debugging not working, or (B) the latter working and the former being degraded somewhat to 1 intermingled console. Thanks for considering.

[gandalfas]

I think that we should look at how the inferior is launched. I suspect that this might be a bug in lldb-server. One quick test would be to try a “remote” (it can be run on the same host) debug session with ds2 (https://github.com/compnerd/ds2).

I will try this next.

[gandalfas]

@compnerd, I see on your ds2 page that I do not have any of the many build tools listed there, and I would really prefer not to install a bunch of things and then uninstall them later, because of all the remnants that will leave.

Might you have a build that I could download and run on my Windows 10 machine? Or, might someone else be able to attempt this?

Thanks.

[compnerd]

The only bits that you should need to install is Visual Studio; Visual Studio already provides CMake + Ninja. The only other dependency is flex and bison which is part of WinFixBison (which is why it is referenced), which doesn't need any installation.

That said ... https://github.com/compnerd/ds2/suites/5881671150/artifacts/198772803 or https://github.com/compnerd/ds2/suites/5881671150/artifacts/198772802 from a recent CI run should foot the bill

[gandalfas]

Thank you for that prebuilt 64-bit binary! That save a lot of time and avoided the post-uninstall crud.

I ran ds2 per the ds2 github page, and have some partial results. You didn't say exactly what you were looking for, so I'll just describe the whole session:

I ran the ds2 server in one command prompt and the lldb client in another. I had server prog crashes at first, suspected the port number, relearned about that, and found that 50000 worked. At repeated trial runs, the failed ones usually caused the port number to be made unavailable and on a subsequent re-use of that port, it caused server ds2 to crash at connection to client time. Going to the next port number worked for a while, but eventually no port number was workable (server always crashed). But one or two tries had some success, shown below.

server window - ds2

-> ds2 gdbserver localhost:50002 "K:\Kim Shared\CodeLab\SwiftTests\hello.build\debug\hello.exe" (waited for client's 'gdb-remote' command) Hello, world! ______ # first print stmt - did not wait at start of prog enter something -> ______ # second print stmt (entered 'abc') End of File was reached - aborting... ______ # final print stmt => readline failed

Notes:

• print to std output worked. readline from std input failed
• only one command prompt existed - the initial one in which ds2 was launched - that was inherited by the debuggee and used for its stdio (in case you were wondering if a second command prompt would be launched)
• obviously no command prompt was needed by lldb user cmds, since it received its commands over TCP
• there was an oddity at end of debuggee execution, where both the debuggee and ?another process? were both reading from std input in the one command prompt. I am not sure how it got into that state, but if it is important, I can reboot and repeat the test.

client window - lldb

-> lldb "K:\Kim Shared\CodeLab\SwiftTests\hello.build\debug\hello.exe" (lldb) target create "K:\Kim Shared\CodeLab\SwiftTests\hello\.build\debug\hello.exe" _____ # output from lldb Current executable set to 'K:\Kim Shared\CodeLab\SwiftTests\hello.build\debug\hello.exe' (x86_64). _____ # output from lldb (lldb) gdb-remote localhost:50002 _____ # my command, per the ds2 web page error: Connection shut down by remote side while waiting for reply to initial handshake packet (lldb)

Notes:

• never really got to send any commands to the server after the gdb-remote command

So, even though I never got remote debugging fully working, was this informative enough to give you clues? If not, I can retry - would you have any thoughts on likely causes for the client error?

[gandalfas]

@compnerd and @adam-fowler,

Was there anything more I can do to help gather information on this?

Or, are we thinking that we can't yet identify which project this should be submitted to? e.g.:

• vadimcn/vscode-lldb - launcher of lldb - is it mishandling the setup of stdio when a custom LLDB is involved?
• apple/swift-lldb - custom LLDB - is it mishandling the redirecting of stdio to the VS Code pane?

If we don't know, should I submit it to vadimcn/vscode-lldb because that component is issuing the errors above, in 'Actual Behavior'? it would at least be a place to start.

Any guidance would be much appreciated. Thanks, Kim

[compnerd]

Hmm, that is interesting. It sounds like even without lldb in the picture, the standard input is ignored? I think that this is sounding more like a Swift runtime issue rather than lldb - it seems like the stream isn't being setup. It may be interesting to a) set a breakpoint on main and see what the stdin handle is setup to b) possibly see if procmon helps see what is going on in the setup?

[compnerd]

BTW, the re-start issue is related to sockets - there is a small timeout associated with sockets that needs to elapse. You can use the -R (or maybe -r?) option to open the socket with SO_REUSEADDR which would avoid that. Any port above 1024 should be available for non-Administrator use IIRC.

[gandalfas]

even without lldb in the picture, the standard input is ignored?

actually, in that case, std in and std out are handled just fine by the program on its own. noted above in "Variations":

Likewise when I run the program directly from the command prompt (no LLDB), the prog uses stdin/stdout just fine in that same command prompt window.

It may be interesting to a) set a breakpoint on main and see what the stdin handle is setup to b) possibly see if procmon helps see what is going on in the setup?

Sure, I will try these.

[gandalfas]

I added these statements to the top of program:

import Foundation // exit()

let myStdIn  = FileHandle.standardInput
let myStdOut = FileHandle.standardOutput
let myStdErr = FileHandle.standardError

print("myStdIn:  \(myStdIn)")
print("myStdOut: \(myStdOut)")
print("myStdErr: \(myStdErr)")

print("Hello, world!")

Running at the cmd prompt (no LLDB) gives this output:

myStdIn:  <FileHandle: 0x00000178d8bc6650>
myStdOut: <FileHandle: 0x00000178d8bcc1a0>
myStdErr: <FileHandle: 0x00000178d8bcbb50>
Hello, world!

Running via VS Code > Run > Start Debugging, and stepping through the 3 initializations, gives this, using 'v myStdIn' (gives same results as hovering over myStdIn):

(Foundation.FileHandle) myStdIn = 0x000001dc814d3f60 {
  Foundation.NSObject = {}
  _handle = 0x354
  _closeOnDealloc = false
  currentBackgroundActivityOwner = nil
  readabilitySource = nil
  writabilitySource = nil
  privateAsyncVariablesLock = 0x000001dc814cc700 {
    Foundation.NSObject = {}
    mutex = 0x1dc814f3c00 {
      pointee = {}
    }
    timeoutCond = 0x1dc814cd180 {
      pointee = {}
    }
    timeoutMutex = 0x1dc814c98d0 {
      pointee = {}
    }
    name = nil
  }
  _queue = nil
  _readabilityHandler = nil
  _writeabilityHandler = nil
}

and 'v myStdOut' gives:

(Foundation.FileHandle) myStdOut = 0x000001dc814c6dd0 {
  Foundation.NSObject = {}
  _handle = 0x364
  _closeOnDealloc = false
  currentBackgroundActivityOwner = nil
  readabilitySource = nil
  writabilitySource = nil
  privateAsyncVariablesLock = 0x000001dc814f5390 {
    Foundation.NSObject = {}
    mutex = 0x1dc814cf2a0 {
      pointee = {}
    }
    timeoutCond = 0x1dc814c3930 {
      pointee = {}
    }
    timeoutMutex = 0x1dc814f4420 {
      pointee = {}
    }
    name = nil
  }
  _queue = nil
  _readabilityHandler = nil
  _writeabilityHandler = nil
}

[compnerd]

It seems reasonable - perhaps compare the value to the handle from C (GetStdHandle(STD_INPUT_HANDLE)).

[gandalfas]

Will do.

Meanwhile, I ran ProcMon and after filtering out all unrelated processes, during the 9 seconds after Start Debugging, I have 15,000 lines, all from swiftc (starting the debug kicks off a quick build to ensure bld is uptodate), swift-build.exe, swift-frontend.exc, codelldb.exe, code.exe, swift.exe, conhost.exe, cmd.exe. A bit of a jungle, not sure which of them is safe to filter out.

[gandalfas]

In short, the Swift handles are equal to the C handles. I added a C DLL to the Swift main prog. Code:

    HANDLE myStdInputHdlFromC = GetStdHandle(STD_INPUT_HANDLE);
    printf("myStdInputHdlFromC: %x\n", myStdInputHdlFromC);

Running the program in the debugger:

Swift:
  myStdIn = 0x0000027c1efd4b90 {    ...   _handle = 0x338       -- using 'v' cmd again
  myStdOut = 0x0000027c1efd31c0 {  ...   _handle = 0x33c

C:
   stdin handle: 0000338 (%x, hex)      -- obtained through hover over variable
   stdout handle: 000033c

I also ran the program at the cmd prompt (no LLDB), but that output was not helpful - all values were nonzero (good), but no handle values are shown in the Swift print stmt. But the above at least shows us the handles are identical in the debug environment.

[compnerd]

Ugh, wait, I wonder if you are running into a "feature" of lldb. One thing that is really annoying about lldb is that it will screw up sometimes with redirection. I've noticed that on Windows, when running with lldb, you will need to switch windows to the lldb input and press the enter key before the inferior will accept any input. I think that should actually make this function properly. If that works, I don't really know how to replicate that with the VSCode setup, but that is likely something to do be done in vscode-lldb.

Note that in my experience, you will also need to do this trick after triggering a breakpoint.

[gandalfas]

OK thanks, I will experiment with that tomorrow morning. But if I remember correctly, when I remove the setting that points to the custom LLDB, the normal LLDB has correct behavior in terms of stdin/stdout. I'll reverify that as well. Have a good night! (if you're in the US)

[adam-fowler]

@Kim4444 when you run your application in lldb can you redirect stdout to a file using

process launch -e stdout.txt -- myApplication.exe

[adam-fowler]

@Kim4444 when you run your application in lldb can you redirect stdout to a file using

process launch -e stdout.txt -- myApplication.exe

Ok I have verified this works as expected

[compnerd]

@adam-fowler I believe that the issue is the input rather than the output. The input is blocked by the lldb instance (which is unblocked by the return), and the inferior will accept input subsequently. I think that the descriptor might be shared?

[adam-fowler]

I've had a look through the vscode-lldb code and it looks like the stdio is setup here https://github.com/vadimcn/vscode-lldb/blob/54d4fbb2dead83e98bd44d07057b67efc4305531/adapter/src/debug_session.rs#L877 From what I can work out (my Rust is rusty, bet nobody made that joke before) it sets stdin to CONIN$ and stdout to CONOUT$. The LLDB SDK docs are not particularly helpful.

[gandalfas]

when running with lldb, you will need to switch windows to the lldb input and press the enter key before the inferior will accept any input.

I was able to reproduce that, using the custom lldb from the cmd prompt, and no redirection at all: if the first cmd I give to lldb is the 'r' cmd or 'b 40', 'r' cmd, such that the readline is encountered before the brkpt, then: text entered in the debuggee window (to the readline) is not echoed to the screen until an lldb cmd is given. and from there on, each window seems unable to get its input until the other window has been given input.

from other tests, in general, whenever one of them is waiting for input, the other does not echo any input typed to it until the first has received the Enter key. somehow the two processes are aware of each other's state. perhaps they are using a mutex to arrange for only one to do input at a time. It works as long as you type into the proper window at the proper time.

That actually makes sense to me, when I think about how a gui debugger works with a gui debuggee - while the debuggee is running, the debugger indicates that, and handles no or few commands. alternatively at a brkpt, when the debugger is in control, the debuggee is not responsive to button clicks.

I don't really know how to replicate that with the VSCode setup

I was also unable to replicate it there. In all tests, no print output appears in the debuggee window or any Terminal pane at bottom of VS Code. As for input: if I try to let the readline happen first, the program reads EOF. If I set a brkpt and enter cmds in the Debug Console tab of VS Code's Terminal pane, to 'kick off' the access to input, then type into the debuggee window (with no echo), the readline still reads EOF instead. I could find no combination of inputting to lldb (using both the lldb prompt in the Debug Console, and gui cmds in the prog source area) that would somehow enable the debuggee to show its print or read its input.

[gandalfas]

(Kim said) But if I remember correctly, when I remove the setting that points to the custom LLDB, the normal LLDB has correct behavior in terms of stdin/stdout. I'll reverify that as well.

I probably reported this above, but for convenience I'll just redescribe it here: I removed the setting, and output from print appeared immediately, statement by statement, in the Terminal tab of the VS Code's bottom panel. When the debugger executes the readline, the 'current line of execution' arrow disappears, and it waits for user input in the Terminal tab. That is echoed immediately, and after hitting Enter, the debugger gets control at the next brkpt. All as expected.

[gandalfas]

so to me, either vscode-lldb takes different setup paths when the lldb.library setting exists, or, custom LLDB is handling the VS Code scenario differently than the standard LLDB.