Upgrade Ruby to a consistent version on all platforms

[lethosor]

Update: See #2081. Our current plan is to remove Ruby support instead.

Different platforms use different precompiled versions of libruby, currently (as of 0.47.04-r3):

• 2.0.0 on win64
• 1.8.7 on win32, osx32, and all Linux
• Whatever is provided by the system on osx64

These were manually built and are somewhat tricky to rebuild (particularly since they need to be built externally), so they are mostly outdated. Changes across different versions can lead to compatibility issues across platforms that users don't expect, like #1613 and https://github.com/DFHack/scripts/pull/227#discussion_r528072627. Update: our current solution also doesn't work on macOS 11, per #1742

Coming up with a CI pipeline to build Ruby with the correct toolchains/flags for DFHack could be a good long-term solution here.

[lethosor]

Work in progress here: https://github.com/lethosor/ruby-build

So far, testing on Linux with Ruby 2.7.2 has run into one issue so far: the active exception seems to not be getting cleared after evaluating code fragments, so running a script that produces an error once will print the same error message on subsequent runs:

[DFHack]# :rb puts RUBY_VERSION
1.8.7
[DFHack]# startdwarf 9
[DFHack]# startdwarf 1
E: RuntimeError: ../../code/dfhack/scripts/startdwarf.rb:16: too low
 ../../code/dfhack/scripts/startdwarf.rb:16
 (eval):18:in `load'
 (eval):18
 (eval):18:in `catch'
 (eval):18
[DFHack]# startdwarf 9
[DFHack]# startdwarf 9

# after upgrading:

[DFHack]# :rb puts RUBY_VERSION
2.7.2
[DFHack]# startdwarf 9
[DFHack]# startdwarf 1
E: RuntimeError: too low
dfhack/scripts/startdwarf.rb:15:in `<top (required)>'
 eval:2:in `load'
 eval:2:in `block in <main>'
 eval:2:in `catch'
 eval:2:in `<main>'
[DFHack]# startdwarf 9
E: RuntimeError: too low
[DFHack]# startdwarf 9
E: RuntimeError: too low

Note that the script is actually running to completion, verified by adding additional puts statements - this is just a display/cleanup issue.

I suspect the issue lies in one of these places: https://github.com/DFHack/dfhack/blob/eb6957a1671bf8db23eb0d08a386b94ceb0c9496/plugins/ruby/ruby.cpp#L487-L491 https://github.com/DFHack/dfhack/blob/eb6957a1671bf8db23eb0d08a386b94ceb0c9496/plugins/ruby/ruby.cpp#L405-L409

[lethosor]

Ruby 2.0.0 (648) and 2.4.10 on Linux do not exhibit the above issue, so it must have been introduced since then.

[lethosor]

Fixed the above issue with 2.7.

Ruby 3.0 has at least 2 major issues:

• Printing tracebacks segfaults (and this happens on startup due to the below error, which is fun!) on his line: https://github.com/DFHack/dfhack/blob/eb6957a1671bf8db23eb0d08a386b94ceb0c9496/plugins/ruby/ruby.cpp#L411 I think this has something to do with the "class" method specifically, because "backtrace" works. Running rb_str_to_cstr(rb_obj_as_string(rb_funcall(err, rb_intern("backtrace"), 0))) in GDB works, and running rb_str_to_cstr(rb_obj_as_string(err)) gives the next error:
• "undefined method '[]' for Dir:Class" at https://github.com/DFHack/dfhack/blob/eb6957a1671bf8db23eb0d08a386b94ceb0c9496/plugins/ruby/ruby.rb#L258 Running this same line in a standalone Ruby 3 interpreter works (replacing require with puts prints the expected files), so I think the issue is due to Dir.[] being implemented in Ruby in Ruby 3 and C in Ruby 2

I don't currently know enough about Ruby to fix these, and Ruby 3 is only a couple days old, so I think we should stick with 2.7 for now, since it's still pretty new (it's only 1 year old for now, and likely supported for around 2 more: https://www.ruby-lang.org/en/downloads/branches/ ).

[lethosor]

Testing Ruby 2.7.2 on Windows: the DFHack console seems to lose all color support (all output from commands like ls -a is a single color).

This is with a Ruby plugin from 0.47.04-r4 - unsure if any changes from my branch would help.

[lethosor]

Testing Ruby 2.7.2 on Windows: the DFHack console seems to lose all color support (all output from commands like ls -a is a single color).

This may be due to ruby_sysinit (Ruby 3 calls SetConsoleMode from there; Ruby 2.7 doesn't, but it could be interfering in some other way). Per https://github.com/ruby/ruby/blob/v2_7_2/ruby.c#L2451-L2456, we may not need to call ruby_sysinit, but we currently are: https://github.com/DFHack/dfhack/blob/92a01e656f816c474fe43a6abb480f01aef38630/plugins/ruby/ruby.cpp#L433-L438

Update: Ruby 1.9 (when we starting calling ruby_sysinit) did not have this comment: https://github.com/ruby/ruby/blob/v1_9_3_551/ruby.c#L1868 However, I don't see why it would have been necessary to call there either. Setting origarg is likely the important bit, but every use of it in ruby.c appears to be internal or checks to make sure it was initialized.

Update: omitting the call to ruby_sysinit on Linux with Ruby 2.7.2 seems to work.

[lethosor]

I'm suspecting that we may need to patch Ruby on Windows. Without ruby_sysinit, it crashes immediately on startup, and with ruby_sysinit, it breaks the console. The alternative is removing Ruby entirely and rewriting all of our Ruby scripts in Lua (there are around 25).

Sadly, the formerly-reproducible builds from https://github.com/lethosor/ruby-build have broken on macOS due to an undetermined dependency change. I was able to work around an autoconf incompatibility by skipping the autoconf step entirely, but now it fails to compile due to a problem with a system header:

gcc-10 -Os -fno-common -pipe  -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fno-strict-overflow -fvisibility=hidden -fexcess-precision=standard -DRUBY_EXPORT -DCANONICALIZATION_FOR_MATHN -I. -I.ext/include/x86_64-darwin19 -I./include -I. -I./enc/unicode/12.1.0 -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT   -DDISABLE_RUBYGEMS=1 -o range.o -c range.c
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Security.framework/Headers/AuthSession.h:32,
                 from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Security.framework/Headers/Security.h:42,
                 from random.c:342:
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Security.framework/Headers/Authorization.h:193:7: error: variably modified 'bytes' at file scope
  193 |  char bytes[kAuthorizationExternalFormLength];
      |       ^~~~~

Some searching suggests that switching to Clang would fix this, but this worked just fine with GCC 10 back in December, and the build is still running on macOS 10.15.

@oevseev pointed me to https://rubies.travis-ci.org/ at https://github.com/DFHack/dfhack/issues/1742#issuecomment-819917573 here (thanks!). This has a collection of Linux and macOS binaries that might work for our needs. I was trying to use toolchains that match DFHack's as closely as possible (hence using GCC on macOS) but that may not be as necessary on these platforms since Ruby is written in C, not C++.

Travis doesn't appear to provide Windows Ruby binaries, but if we need to patch those anyway, maybe that's okay. I'll also probably try to find another Windows binary in case ours is just broken. I believe our 2.0.0 Windows libruby came from https://rubyinstaller.org/ directly.

[jandr]

Fixed the above issue with 2.7.

I tried using Ruby shipped with Debian (package ruby2.7-dev has libruby-2.7.so), as I am trying to build a Debian package for DFHack, and I still seem to have the issue there:

[DFHack]# :rb puts RUBY_VERSION
2.7.4
[DFHack]# startdwarf 9
[DFHack]# startdwarf 1
E: RuntimeError: too low
 hack/scripts/startdwarf.rb:15:in `<top (required)>'
 eval:2:in `load'
 eval:2:in `block in <main>'
 eval:2:in `catch'
 eval:2:in `<main>'
[DFHack]# startdwarf 9
E: RuntimeError: too low
[DFHack]# startdwarf 9
E: RuntimeError: too low

[lethosor]

Did you make the necessary linked change (67e91293a31e0e326b04708113c947aa200d379f)? It's not on develop because Ruby 2.7 support is not ready, per the rest of this thread.

[jandr]

Sorry, I had completely missed the linked change not being part of the branch I was using.

After applying the changes from https://github.com/DFHack/dfhack/commit/67e91293a31e0e326b04708113c947aa200d379f the issue is no longer there.

Thanks!

[lethosor]

Update: we plan to eventually remove Ruby support because this issue has proven too difficult to address: #2081