Compile support libraries with Position Independent Code (PIC)

[luislavena]

Attempting to use Crystal through tools like asdf might result in compilation errors triggered by embedded libraries not being compiled with -fPIC:

/usr/bin/ld: /home/luis/.asdf/installs/crystal/0.20.5/embedded/lib/../lib/libyaml.a(api.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /home/luis/.asdf/installs/crystal/0.20.5/embedded/lib/../lib/libyaml.a(parser.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /home/luis/.asdf/installs/crystal/0.20.5/embedded/lib/../lib/libyaml.a(scanner.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /home/luis/.asdf/installs/crystal/0.20.5/embedded/lib/../lib/libyaml.a(reader.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status

I wasn't able to check the build options used for the embedded libraries, but wanted to create the issue before looks like asdf gain more traction and we get more reports like this.

Thank you. :heart: :heart: :heart:

[mverzilli]

Absolutely newbie question:

Is this -fPIC flag something that we can safely add to the build configuration or do you know of any downsides? The documentation starts with "If supported by the target machine...", which tempts me into just adding it and letting the compiler decide how to emit code for each specific target machine.

[luislavena]

Hello @mverzilli, considering omnibus package only works on x86 and x86_64 platforms, usage of -fPIC will have no downside.

The usage of the flag helps out avoid exhausting the global offset table which can happen when the size of the linked objects are bigger than supported by the platform or default settings.

The special thing is portability, since the static library bundled in the package is produced by a combination of OS/kernel version that might differ from the target when executing, plus the unknown scenario where a shared library might be also interacting with the static library.

Things like libtool help alleviate the offset position of the objects in the static library but we don't rely on libtool.

I believe this StackOverflow thread covers some of these details.

Cheers.

[matiasgarciaisaia]

@luislavena I'm still trying to understand about Crystal's build process, so help me a bit with this.

I'm seeing that -fPIC is getting added to the CFLAGS envvar during the build. I'm not sure how that affects the resulting binary - or even if that's platform-dependent, as Homebrew Crystal is built differently.

Should that work, or should we add the -fPIC in a different phase of the compilation?

[luislavena]

Hmmm, :thinking:

I just downloaded 0.21.1-1 binaries from releases page:

$ tar -xf crystal-0.21.1-1-linux-x86_64.tar.gz
...
$ cd crystal-0.21.1-1/

$ export PATH=`pwd`/bin:$PATH

$ crystal --version
Crystal 0.21.1 [3c6c75e] (2017-03-06) LLVM 3.5.0

$ bin/crystal env
CRYSTAL_CACHE_DIR="/home/luis/.cache/crystal"
CRYSTAL_PATH="lib:/home/luis/code/_experiments/crystal-0.21.1-1/src"
CRYSTAL_VERSION="0.21.1"

With that compiler and no compiler installed in the system (in /opt/crystal) the following happens when attempt to use something like YAML:

$ cat test.cr 
require "yaml"

p YAML.parse("foo:\n- 1\n- 2\n")

$ bin/crystal build test.cr 
/usr/bin/ld: /home/luis/code/_experiments/crystal-0.21.1-1/embedded/lib/../lib/libyaml.a(api.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /home/luis/code/_experiments/crystal-0.21.1-1/embedded/lib/../lib/libyaml.a(parser.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /home/luis/code/_experiments/crystal-0.21.1-1/embedded/lib/../lib/libyaml.a(scanner.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /home/luis/code/_experiments/crystal-0.21.1-1/embedded/lib/../lib/libyaml.a(reader.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
Error: execution of command failed with code: 1: `cc -o "/home/luis/code/_experiments/crystal-0.21.1-1/test" "${@}"  -rdynamic  -lyaml -lpcre -lgc -lpthread /home/luis/code/_experiments/crystal-0.21.1-1/src/ext/libcrystal.a -levent -lrt -ldl -L/usr/lib -L/usr/local/lib`

But that doesn't happen when using the debian package (not sure if that information helps).

I was not able to use the vagrant box to test things out (keep failing for me) but will love to help figure this out.

Thank you.

[luislavena]

Perhaps it requires to set CFLAGS on all the packages and not just crystal one?

[matiasgarciaisaia]

The release workflow is not as streamlined as we'd like to. It may very well be that we're bundling different compiler builds in each of the distribution channels - well, as a matter of fact, we are.

It's definitely something I'd like to work on in the short/mid-term. I'll make sure to take this issue into account then 👍