Connection refused error after updating to 1.6.X in a Rails project running inside docker

[ralph]

Your environment

• ruby -v: ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [aarch64-linux]
• rdbg -v: working: rdbg 1.5.0, non-working rdbg 1.6.X

Describe the bug I'm using the official Ruby 3.1.2-slim docker image. After updating the debug gem from 1.5.0 to 1.6.X, I cannot attach to the running process any more.

I'm running a Rails project with the latest Rails version (7.0.4) and the Puma web server. This is the command I'm running:

rdbg --open --nonstop -c -- rails s -b '0.0.0.0'

Using debug 1.5.0, all is well:

❯ dc exec web rdbg -A
# No sourcefile available for /usr/local/bundle/gems/puma-5.6.5/lib/puma/single.rb
=>#0    [C] Thread#join at /usr/local/bundle/gems/puma-5.6.5/lib/puma/single.rb:61
  #1    Puma::Single#run at /usr/local/bundle/gems/puma-5.6.5/lib/puma/single.rb:61
  # and 15 frames (use `bt' command for all frames)

Stop by SIGURG
(rdbg:remote) c    # continue command

Socket name (automatically chosen by the debug gem): /tmp/ruby-debug-sock-0/ruby-debug-ruby-debug-1

Using debug 1.6.X, the connection does not work any more:

/usr/local/lib/ruby/3.1.0/socket.rb:64:in `connect': Connection refused - connect(2) for /tmp/ruby-debug-sock-0/ruby-debug-UnknownUser-1 (Errno::ECONNREFUSED)
        from /usr/local/lib/ruby/3.1.0/socket.rb:64:in `connect_internal'
        from /usr/local/lib/ruby/3.1.0/socket.rb:137:in `connect'
        from /usr/local/lib/ruby/3.1.0/socket.rb:1078:in `unix'
        from /usr/local/bundle/gems/debug-1.6.0/lib/debug/client.rb:158:in `connect_unix'
        from /usr/local/bundle/gems/debug-1.6.0/lib/debug/client.rb:108:in `initialize'
        from /usr/local/bundle/gems/debug-1.6.0/exe/rdbg:31:in `new'
        from /usr/local/bundle/gems/debug-1.6.0/exe/rdbg:31:in `<top (required)>'
        from /app/bin/rdbg:29:in `load'
        from /app/bin/rdbg:29:in `<main>'

Socket name (automatically chosen by the debug gem): /tmp/ruby-debug-sock-0/ruby-debug-UnknownUser-1

Notice the change in the socket name, maybe this has something to do with this issue? The socket does exist, though, and permissions seem correct.

To Reproduce Update debug to 1.6.X. I've tried all released 1.6 versions, none of them is working. Going back to 1.5 resolves the issue for me.

Expected behavior rdbg -A connects to the running process.

[ko1]

I need to repro this issue on my machine for debugging. Can you write step by step guide with docker container?

[ralph]

Hi @ko1, thank you for your reply. I added a simple Rails app for reproduction here:

https://github.com/ralph/ruby-debug-761

I tried using the official docker guide (where not outdated) to be as standards compliant as possible: https://docs.docker.com/samples/rails/

You should be able to start the server by running dc build followed by dc up. Note dc: aliased to docker compose.

Start command is here: https://github.com/ralph/ruby-debug-761/blob/main/docker-compose.yml#L15

I try to connect to the debugger like this:

dc exec web rdbg -A
/usr/local/lib/ruby/3.1.0/socket.rb:64:in `connect': Connection refused - connect(2) for /tmp/ruby-debug-sock-0/ruby-debug-UnknownUser-1 (Errno::ECONNREFUSED)
        from /usr/local/lib/ruby/3.1.0/socket.rb:64:in `connect_internal'
        from /usr/local/lib/ruby/3.1.0/socket.rb:137:in `connect'
        from /usr/local/lib/ruby/3.1.0/socket.rb:1078:in `unix'
        from /usr/local/bundle/gems/debug-1.6.2/lib/debug/client.rb:158:in `connect_unix'
        from /usr/local/bundle/gems/debug-1.6.2/lib/debug/client.rb:108:in `initialize'
        from /usr/local/bundle/gems/debug-1.6.2/exe/rdbg:31:in `new'
        from /usr/local/bundle/gems/debug-1.6.2/exe/rdbg:31:in `<top (required)>'
        from /usr/local/bundle/bin/rdbg:25:in `load'
        from /usr/local/bundle/bin/rdbg:25:in `<main>'

I also added a branch with debug 1.5 installed, which works: https://github.com/ralph/ruby-debug-761/tree/ruby-debug-1-5

dc exec web rdbg -A
# No sourcefile available for /usr/local/bundle/gems/puma-5.6.5/lib/puma/single.rb
=>#0    [C] Thread#join at /usr/local/bundle/gems/puma-5.6.5/lib/puma/single.rb:61
  #1    Puma::Single#run at /usr/local/bundle/gems/puma-5.6.5/lib/puma/single.rb:61
  # and 16 frames (use `bt' command for all frames)

Stop by SIGURG
(rdbg:remote) c    # continue command

[jroblesap]

Could @need_pause_at_first have something to do with this?

[jroblesap]

Or perhaps https://github.com/ruby/debug/blob/master/lib/debug/server_dap.rb#L135

[jroblesap]

@ralph @ko1 it actually works without the -n flag but then it's unusable.

Also the following issues share the same root cause as this one

• https://github.com/ruby/debug/issues/368
• https://github.com/connorshea/vscode-ruby-test-adapter/issues/92#issuecomment-1191477785

I am using

• Ruby 2.7.6
• Rails 6.1
• Puma 5.0
• debug 1.7.0dev (https://github.com/ruby/debug/commit/e422abe4cae49767ec3c17af010f0dff1b2c6dfa)

I am connecting with TCP/IP via Docker.

[ralph]

Hi, I was able to make my own sample repository work with 1.6.X with only minimal changes: https://github.com/ralph/ruby-debug-761/pull/1/files

Problem is, I don't understand why this is working, while the previous version was not. Adding back bundle exec breaks 1.6 again. So does removing the $PATH. If someone is able to figure this out, I'd be very thankful.

[ralph]

I added another variant here, without tweaking $PATH, using the binstub directly: https://github.com/ralph/ruby-debug-761/pull/2/files

It works, too, but I still don't quite understand why the previous version of the command worked with 1.5, while 1.6 does not work.

[jroblesap]

Thanks for this @ralph! I can also confirm using binstub works on 1.6.x

However, I still see the EADDRINUSE flood from 1.5.x:

/usr/local/lib/ruby/2.7.0/socket.rb:201:in `bind': Address already in use - bind(2) for 0.0.0.0:1234 (Errno::EADDRINUSE)
DEBUGGER[bin/rails#2518]: Attaching after process 44 fork to child process 2518
DEBUGGER[bin/rails#2518]: Connected.
DEBUGGER[bin/rails#2518]: ReaderThreadError: no implicit conversion of nil into IO
[[44] - Worker 0 (PID: 2518) booted in 0.0s, phase: 0
"/usr/local/bundle/ruby/2.7.0/gems/debug-1.6.3/lib/debug/server.rb:178:in `select'",
 "/usr/local/bundle/ruby/2.7.0/gems/debug-1.6.3/lib/debug/server.rb:178:in `process'",
 "/usr/local/bundle/ruby/2.7.0/gems/debug-1.6.3/lib/debug/server.rb:73:in `block (3 levels) in activate'",
 "/usr/local/bundle/ruby/2.7.0/gems/debug-1.6.3/lib/debug/server.rb:269:in `setup_interrupt'",
 "/usr/local/bundle/ruby/2.7.0/gems/debug-1.6.3/lib/debug/server.rb:71:in `block (2 levels) in activate'",
 "/usr/local/bundle/ruby/2.7.0/gems/debug-1.6.3/lib/debug/server.rb:34:in `accept'",
 "/usr/local/bundle/ruby/2.7.0/gems/debug-1.6.3/lib/debug/server.rb:417:in `accept'",
 "/usr/local/bundle/ruby/2.7.0/gems/debug-1.6.3/lib/debug/server.rb:49:in `block in activate'"]
DEBUGGER[puma: cluster worker 0: 44 [app]#2518]: Disconnected.
DEBUGGER[bin/rails#2533]: Attaching after process 44 fork to child process 2533
DEBUGGER[bin/rails#2533]: Connected.
DEBUGGER[bin/rails#2533]: ReaderThreadError: no implicit conversion of nil into IO
["/usr/local/bundle/ruby/2.7.0/gems/debug-1.6.3/lib/debug/server.rb:178:in `select'"[44] - Worker 2 (PID: 2533) booted in 0.0s, phase: 0
,
 "/usr/local/bundle/ruby/2.7.0/gems/debug-1.6.3/lib/debug/server.rb:178:in `process'",
 "/usr/local/bundle/ruby/2.7.0/gems/debug-1.6.3/lib/debug/server.rb:73:in `block (3 levels) in activate'",
 "/usr/local/bundle/ruby/2.7.0/gems/debug-1.6.3/lib/debug/server.rb:269:in `setup_interrupt'",
 "/usr/local/bundle/ruby/2.7.0/gems/debug-1.6.3/lib/debug/server.rb:71:in `block (2 levels) in activate'",
 "/usr/local/bundle/ruby/2.7.0/gems/debug-1.6.3/lib/debug/server.rb:34:in `accept'",
 "/usr/local/bundle/ruby/2.7.0/gems/debug-1.6.3/lib/debug/server.rb:417:in `accept'",
 "/usr/local/bundle/ruby/2.7.0/gems/debug-1.6.3/lib/debug/server.rb:49:in `block in activate'"]
DEBUGGER[puma: cluster worker 2: 44 [app]#2533]: Disconnected.
DEBUGGER[bin/rails#2543]: Attaching after process 44 fork to child process 2543

I haven't been able to pinpoint if the issue is with Rails, Puma, Ruby or debug.

[ralph]

Sorry, I can't help with that. I'm using the socket, not TCP.

[jroblesap]

@ralph okay! Looks like the socket is a bit more reliable than TCP.

Have you tried remote debugging (maybe with VSCode) using UDS?

[ralph]

No, I'm doing it the old school way (with rdbg -A). Sorry!

[nilbus]

I'm experiencing this same error with debug-1.7.0, also in a Docker container, both with Socket and TCP connections.

A couple clues:

1. I do reproduce this with require 'debug/open_nonstop' and DEBUGGER__.open(nonstop: true), but unlike @ralph's experience, it does work when I start the Rails server with rdbg --open --nonstop -c -- bundle exec rails s -b 0.0.0.0.
2. With TCP connections, the port does not get open.
3. When not working, the socket file exists, but connections to it are refused.

root@8c4881cfbc82:/app# rdbg -A
Traceback (most recent call last):
        12: from /usr/local/bundle/bin/rdbg:23:in `<main>'
        11: from /usr/local/bundle/bin/rdbg:23:in `load'
        10: from /usr/local/bundle/gems/debug-1.7.0/exe/rdbg:46:in `<top (required)>'
         9: from /usr/local/bundle/gems/debug-1.7.0/exe/rdbg:46:in `new'
         8: from /usr/local/bundle/gems/debug-1.7.0/lib/debug/client.rb:125:in `initialize'
         7: from /usr/local/bundle/gems/debug-1.7.0/lib/debug/client.rb:168:in `connect_unix'
         6: from /usr/local/bundle/gems/debug-1.7.0/lib/debug/client.rb:103:in `list_connections'
         5: from /usr/local/bundle/gems/debug-1.7.0/lib/debug/client.rb:103:in `map'
         4: from /usr/local/bundle/gems/debug-1.7.0/lib/debug/client.rb:104:in `block in list_connections'
         3: from /usr/local/lib/ruby/2.7.0/socket.rb:1078:in `unix'
         2: from /usr/local/lib/ruby/2.7.0/socket.rb:137:in `connect'
         1: from /usr/local/lib/ruby/2.7.0/socket.rb:64:in `connect_internal'
/usr/local/lib/ruby/2.7.0/socket.rb:64:in `connect': Connection refused - connect(2) for /tmp/ruby-debug-sock-0/ruby-debug-UnknownUser-28 (Errno::ECONNREFUSED)

root@8c4881cfbc82:/app# file /tmp/ruby-debug-sock-0/ruby-debug-UnknownUser-28
/tmp/ruby-debug-sock-0/ruby-debug-UnknownUser-28: socket

[ko1]

@nilbus Thank you for your report. Could you make a step by step repro process?

[mschoenlaub]

Exactly same situation here, btw.

[seankibler]

I was able to eliminate the EADDRINUSE flood by setting Puma workers to 0 (disable cluster mode) with TCP.

I'm going to do some more testing but I wonder if it has to do with the default behavior of RUBY_DEBUG_FORK_MODE being to init debugger for both parent and child, with Puma in cluster mode (workers >= 1) I could see this being the cause.