VM::init_loadpath won't work with static builds

[danielpclark]

When choosing to compile with RUBY_STATIC flag the test for class::vm::VM::init_loadpath fails.

[danielpclark]

Fixing this may fix #44

[felix-d]

This is the failures I get when running tests on my machine with a static build.

failures:

---- src/class/string.rs - class::string::RString::is_valid_encoding (line 530) stdout ----
thread 'src/class/string.rs - class::string::RString::is_valid_encoding (line 530)' panicked at 'test executable failed:

dyld: lazy symbol binding failed: Symbol not found: _rb_encdb_declare
  Referenced from: /opt/rubies/ruby-2.5.3/lib/ruby/2.5.0/x86_64-darwin18/enc/encdb.bundle
  Expected in: flat namespace

dyld: Symbol not found: _rb_encdb_declare
  Referenced from: /opt/rubies/ruby-2.5.3/lib/ruby/2.5.0/x86_64-darwin18/enc/encdb.bundle
  Expected in: flat namespace


', librustdoc/test.rs:367:17
note: Run with `RUST_BACKTRACE=1` for a backtrace.

---- src/class/string.rs - class::string::RString::from_bytes (line 110) stdout ----
thread 'src/class/string.rs - class::string::RString::from_bytes (line 110)' panicked at 'test executable failed:

dyld: lazy symbol binding failed: Symbol not found: _rb_encdb_declare
  Referenced from: /opt/rubies/ruby-2.5.3/lib/ruby/2.5.0/x86_64-darwin18/enc/encdb.bundle
  Expected in: flat namespace

dyld: Symbol not found: _rb_encdb_declare
  Referenced from: /opt/rubies/ruby-2.5.3/lib/ruby/2.5.0/x86_64-darwin18/enc/encdb.bundle
  Expected in: flat namespace


', librustdoc/test.rs:367:17

---- src/class/vm.rs - class::vm::VM::init_loadpath (line 53) stdout ----
thread 'src/class/vm.rs - class::vm::VM::init_loadpath (line 53)' panicked at 'test executable failed:

dyld: lazy symbol binding failed: Symbol not found: _rb_encdb_declare
  Referenced from: /opt/rubies/ruby-2.5.3/lib/ruby/2.5.0/x86_64-darwin18/enc/encdb.bundle
  Expected in: flat namespace

dyld: Symbol not found: _rb_encdb_declare
  Referenced from: /opt/rubies/ruby-2.5.3/lib/ruby/2.5.0/x86_64-darwin18/enc/encdb.bundle
  Expected in: flat namespace


', librustdoc/test.rs:367:17


failures:
    src/class/string.rs - class::string::RString::from_bytes (line 110)
    src/class/string.rs - class::string::RString::is_valid_encoding (line 530)
    src/class/vm.rs - class::vm::VM::init_loadpath (line 53)

test result: FAILED. 181 passed; 3 failed; 3 ignored; 0 measured; 0 filtered out

[felix-d]

The problem is not related to the particular missing symbol mentioned in the stacktrace above. Actually, any VM::require call to require a .bundle (osx) or .so (linux) will raise an exception for missing rb_... symbols. I'm not super familiar with linking concepts, but I guess we're missing a link somewhere or using bundles that require a shared libruby.

[danielpclark]

The Ruby programming language's source code decides which symbol names to export publicly and which ones remain private. Sometimes these are even OS specific as can be seen from #49 .

rb_encdb_declare is defined here: Ruby - ruby/encoding.c#L353 and used in the internal header here Ruby - ruby/internal.h#L1410 which means that we won't have access to it. The text encoding database is either available to be included, or included, by calling ruby_init_loadpath which is available publicly and that method calls out to ruby_init_loadpath_safe here Ruby - ruby/ruby.c#L562 in which you can see has many C preprocessor conditionals which I suspect is causing this issue for static linking for us.

Without the ruby_init_loadpath giving us access to the source code available from Ruby - ruby/encoding.c we have only very limited use of text encodings which is also somewhat OS dependent and I believe typically includes the following from miniinit.c.

void
Init_enc(void)
{
    rb_encdb_declare("ASCII-8BIT");
    rb_encdb_declare("US-ASCII");
    rb_encdb_declare("UTF-8");
    rb_encdb_alias("BINARY", "ASCII-8BIT");
    rb_encdb_alias("ASCII", "US-ASCII");
}

We could re-implement the code that we've been excluded from in Rust for static linking but that may be a lot of work. If this were done we'd conditionally compile this code based on the RUBY_STATIC env.

Basically we need to get what isn't working in our loadpath during static builds to work. There's no telling how simple or difficult this is with my current knowledge of Ruby's C code and C programming in general.

[felix-d]

Thanks for the writeup. My knowledge of Ruby internals and build process is quite limited as well, but this makes sense and definitely help me better understand the issue. I’ll keep investigating and keep you posted if I find anything.

[felix-d]

Something I've noticed: if I try to require another lib such as zlib, I get

dyld: lazy symbol binding failed: Symbol not found: _rb_define_attr
  Referenced from: /opt/rubies/2.3.3/lib/ruby/2.3.0/x86_64-darwin16/zlib.bundle
  Expected in: flat namespace

However, rb_define_attr is not part of internal.h but ruby.h 🤔

[danielpclark]

Something I've noticed: if I try to require another lib such as zlib

Did you make sure the flags you passed in were the absolute last item printed from build.rs ? Because if the flags are early it messes things up.

Also Ruby already includes zlib with the default flags when build.rs calls the "LIBS" option from rbconfig.

[felix-d]

I may be missing something. I did not change build.rs to pass new flags. Also RbConfig::CONFIG['LIBS'] returns -lpthread -lgmp -ldl -lobjc on my machine, if that's what you are referring to.

What I'm alluding to is these failures on the static builds when trying to require("zlib").

This is just an example to illustrate that it might not be related to internal.h as you mentioned, unless I'm mistaken somewhere, which is strongly possible.

[danielpclark]

No you're right. I was mistaken. LIBS comes out to -lpthread -lgmp -ldl -lcrypt -lm on my machine.

[sandstrom]

Would it be helpful to look at how Helix solved this? (if they ever did)

https://github.com/tildeio/helix

My (limited) understanding is that Helix tried to solve basically the same problem that Rutie does. So maybe one could borrow techniques from them.

Or maybe reach out to @wagenet or @chancancode and see if they have any advice.

---

If anyone would be interested in working on this, we can sponsor the open-source work (via Github sponsors) of making rutie work with static builds. Ping me in this thread if you're interested.