charlock_holmes Build fails to link `-licuuc -lz` on gentoo

gem install fails with these build errors (and warnings)

host ~ # gem install charlock_holmes
Building native extensions.  This could take a while...
ERROR:  Error installing charlock_holmes:
        ERROR: Failed to build gem native extension.

        /usr/bin/ruby19 extconf.rb
checking for main() in -licui18n... yes
checking for main() in -licui18n... yes
checking for unicode/ucnv.h... yes
  -- tar zxvf file-5.08.tar.gz
  -- ./configure --prefix=/usr/local/lib64/ruby/gems/1.9.1/gems/charlock_holmes-0.6.9.4/ext/charlock_holmes/dst/ --disable-shared --enable-static --with-pic
  -- patch -p0 < ../file-soft-check.patch
  -- make -C src install
  -- make -C magic install
checking for main() in -lmagic_ext... yes
checking for magic.h... yes
creating Makefile

make
compiling encoding_detector.c
common.h:14:14: warning: 'charlock_new_enc_str' defined but not used [-Wunused-function]
compiling converter.c
common.h:23:14: warning: 'charlock_new_str' defined but not used [-Wunused-function]
common.h:32:14: warning: 'charlock_new_str2' defined but not used [-Wunused-function]
compiling ext.c
common.h:14:14: warning: 'charlock_new_enc_str' defined but not used [-Wunused-function]
common.h:23:14: warning: 'charlock_new_str' defined but not used [-Wunused-function]
common.h:32:14: warning: 'charlock_new_str2' defined but not used [-Wunused-function]
compiling transliterator.cpp
common.h:14:14: warning: 'VALUE charlock_new_enc_str(const char*, size_t, void*)' defined but not used [-Wunused-function]
common.h:32:14: warning: 'VALUE charlock_new_str2(const char*)' defined but not used [-Wunused-function]
linking shared-object charlock_holmes/charlock_holmes.so
encoding_detector.o: In function `rb_get_supported_encodings':
encoding_detector.c:(.text+0xa0): undefined reference to `uenum_count_49'
encoding_detector.c:(.text+0xc1): undefined reference to `uenum_next_49'
encoding_detector.c:(.text+0x12a): undefined reference to `uenum_next_49'
encoding_detector.c:(.text+0x16c): undefined reference to `uenum_next_49'
encoding_detector.c:(.text+0x1ad): undefined reference to `uenum_next_49'
encoding_detector.c:(.text+0x1f7): undefined reference to `uenum_next_49'
encoding_detector.o:encoding_detector.c:(.text+0x234): more undefined references to `uenum_next_49' follow
encoding_detector.o: In function `rb_encdec__alloc':
encoding_detector.c:(.text+0x75d): undefined reference to `u_errorName_49'
converter.o: In function `rb_converter_convert':
converter.c:(.text+0xb1): undefined reference to `ucnv_convert_49'
converter.c:(.text+0xf1): undefined reference to `ucnv_convert_49'
converter.c:(.text+0x18a): undefined reference to `u_errorName_49'
transliterator.o: In function `rb_transliterator_id_list':
transliterator.cpp:(.text+0x15a): undefined reference to `u_errorName_49'
transliterator.o: In function `rb_transliterator_transliterate':
transliterator.cpp:(.text+0x242): undefined reference to `icu_49::UnicodeString::UnicodeString(char const*, int)'
transliterator.cpp:(.text+0x26e): undefined reference to `icu_49::UnicodeString::~UnicodeString()'
transliterator.cpp:(.text+0x287): undefined reference to `icu_49::UMemory::operator new(unsigned long)'
transliterator.cpp:(.text+0x29d): undefined reference to `icu_49::UnicodeString::UnicodeString(char const*, int)'
transliterator.cpp:(.text+0x2e5): undefined reference to `icu_49::UnicodeString::toUTF8(icu_49::ByteSink&) const'
transliterator.cpp:(.text+0x339): undefined reference to `icu_49::ByteSink::~ByteSink()'
transliterator.cpp:(.text+0x396): undefined reference to `u_errorName_49'
transliterator.cpp:(.text+0x40b): undefined reference to `icu_49::UMemory::operator delete(void*)'
transliterator.cpp:(.text+0x442): undefined reference to `icu_49::UnicodeString::~UnicodeString()'
transliterator.o: In function `icu_49::StringByteSink<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >::~StringByteSink()':
transliterator.cpp:(.text._ZN6icu_4914StringByteSinkISsED2Ev[_ZN6icu_4914StringByteSinkISsED5Ev]+0xf): undefined reference to `icu_49::ByteSink::~ByteSink()'
transliterator.o: In function `icu_49::StringByteSink<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >::~StringByteSink()':
transliterator.cpp:(.text._ZN6icu_4914StringByteSinkISsED0Ev[_ZN6icu_4914StringByteSinkISsED5Ev]+0x13): undefined reference to `icu_49::ByteSink::~ByteSink()'
transliterator.cpp:(.text._ZN6icu_4914StringByteSinkISsED0Ev[_ZN6icu_4914StringByteSinkISsED5Ev]+0x1c): undefined reference to `icu_49::UMemory::operator delete(void*)'
transliterator.o:(.data.rel.ro._ZTVN6icu_4914StringByteSinkISsEE[vtable for icu_49::StringByteSink<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >]+0x28): undefined reference to `icu_49::ByteSink::GetAppendBuffer(int, int, char*, int, int*)'
transliterator.o:(.data.rel.ro._ZTVN6icu_4914StringByteSinkISsEE[vtable for icu_49::StringByteSink<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >]+0x30): undefined reference to `icu_49::ByteSink::Flush()'
transliterator.o:(.data.rel.ro._ZTIN6icu_4914StringByteSinkISsEE[typeinfo for icu_49::StringByteSink<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >]+0x10): undefined reference to `typeinfo for icu_49::ByteSink'
./libmagic_ext.a(compress.o): In function `uncompressgzipped':
/usr/local/lib64/ruby/gems/1.9.1/gems/charlock_holmes-0.6.9.4/ext/charlock_holmes/src/file-5.08/src/compress.c:357: undefined reference to `inflateInit2_'
/usr/local/lib64/ruby/gems/1.9.1/gems/charlock_holmes-0.6.9.4/ext/charlock_holmes/src/file-5.08/src/compress.c:363: undefined reference to `inflate'
/usr/local/lib64/ruby/gems/1.9.1/gems/charlock_holmes-0.6.9.4/ext/charlock_holmes/src/file-5.08/src/compress.c:370: undefined reference to `inflateEnd'
collect2: ld returned 1 exit status
make: *** [charlock_holmes.so] Error 1


Gem files will remain installed in /usr/local/lib64/ruby/gems/1.9.1/gems/charlock_holmes-0.6.9.4 for inspection.
Results logged to /usr/local/lib64/ruby/gems/1.9.1/gems/charlock_holmes-0.6.9.4/ext/charlock_holmes/gem_make.out

I have also tried with --with-icu-dir to no avail.

Also see https://github.com/gitlabhq/gitlabhq/issues/1972

Apr 04 '13 10:04 cinderblock

I suspect the issue is exactly what @elestedt suggested.

"It includes libicuu18n twice"

checking for main() in -licui18n... yes
checking for main() in -licui18n... yes

-licuuc -lz seem to be the unlinked deps

Apr 04 '13 11:04 cinderblock

What version of icu are you using?

Apr 04 '13 22:04 brianmario

/usr/lib64/icu/current -> 49.1.2

Apr 04 '13 22:04 cinderblock

@cinderblock were you able to verify that linking libicuuc worked?

Apr 04 '13 22:04 brianmario

Yes.

This does not fail (because of libs) when linking.

host ~ $ gcc -licui18n -licuuc -lz
/usr/lib/gcc/x86_64-pc-linux-gnu/4.6.3/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status

I know the libs are linking properly because trying to link a fake lib produces a different error:

host ~ $ gcc -lfoobar
/usr/lib/gcc/x86_64-pc-linux-gnu/4.6.3/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find -lfoobar
collect2: ld returned 1 exit status

Apr 05 '13 07:04 cinderblock

Is there a chance this is related to #27?

Apr 08 '13 19:04 cinderblock

Don't think so. Looks like it build in that case - only wrong install path

Apr 08 '13 19:04 elestedt

I guessed as much.

Is there a way to turn on a more verbose debug and see why the needed deps aren't getting linked?

Apr 08 '13 21:04 cinderblock

The following excerpt is from the generated Makefile. It shows that icui18n is linked twice and icuuc and z not linked.

LIBS = $(LIBRUBYARG_SHARED) -lmagic_ext -licui18n -licui18n -lpthread -lrt -ldl -lcrypt -lm -lc

Apr 10 '13 06:04 cinderblock

If I edit that line to add the missing libs and make, it builds the lib correctly.

I'm at a loss of how to add extra missing libs to the build. I've been trying to set environment variables to no avail.

Apr 10 '13 07:04 cinderblock

I don't know enough about the autotools to help much either. cmake is more of my thing.

Apr 10 '13 10:04 elestedt

I found a fork that claims to be for gentoo. See above reference.

I believe the change was simply adding these lines to ext/charlock_holmes/extconf.rb

# Fix underlinking
have_library 'z' or abort 'libz missing'
have_library 'icuuc' or abort 'libicuuc missing'
have_library 'icudata' or abort 'libicudata missing'

How do I gem install a local dir!?

Apr 10 '13 18:04 cinderblock

I was finally able to build this on my system. The the missing lines from above are what did it. (With the exception of icudata which seems to be unnecessary)

In case it helps anyone, I made my edits to exconf.rb and ran gem build charlock_holmes.gemspec (which needed to be modified to remove the non-ascii charater in one of the author's names). That generated charlock_holmes-0.6.9.4.gem which I installed with gem install charlock_holmes-0.6.9.4.gem.

@brianmario, would you like/accept a pull request with these added lines?

Apr 12 '13 21:04 cinderblock

$ nano -w /usr/lib/ruby/1.9.1/x86_64-linux/rbconfig.rb

Find the line below (hint: search for “undefined”, it will be the 1st result) and remove the last ldflag: CONFIG["LDFLAGS"] = "-L. -Wl,-O1 -Wl,--as-needed -rdynamic -Wl,-export-dynamic -Wl,--no-undefined"

Change to: CONFIG["LDFLAGS"] = "-L. -Wl,-O1 -Wl,--as-needed -rdynamic -Wl,-export-dynamic"

$ gem install charlock_holmes

good to go ;)

Apr 13 '13 22:04 tsubus

Hmmm, interesting find @tsubus. That works for me.

Are LDFLAGS set differently on other distros? Is this a gentoo bug?

Apr 14 '13 02:04 cinderblock

This may help as well:

https://github.com/brianmario/charlock_holmes/issues/10#issuecomment-5952544

If this gets 'fixed' to work with gentoo then I can do away with my hacked repo that just has the above patch applied. Note that I didn't write the patch, I'm just a lowly user :)

Apr 22 '13 17:04 travisghansen

I will also note that doing this:

https://github.com/brianmario/charlock_holmes/issues/10#issuecomment-11899472

is not really (by my understanding) an option for gentoo as a whole. But I don't claim to understand any of the reasons.

Apr 22 '13 18:04 travisghansen

I've posted about this on Gentoo's bugzilla. One user there claims this is a desired behavior as it forces developers to explicitly link all needed dependencies.

https://bugs.gentoo.org/show_bug.cgi?id=466838

@brianmario Any chance the missing dependencies can be added to extconf.rb?

Apr 24 '13 19:04 cinderblock

I agree with them on that. If it's a dependency, it should be explicitly linked to... How else can you be certain that it works on all platforms it should build on, when the correct version(s) of the dependencies are installed?

Apr 24 '13 20:04 elestedt

just an FYI, when installing the Gem, rather than editing Rbconfig, you can just pass the desired LDFLAGS on the commandline like so:

gem install charlock_holmes -- --with-ldflags='-L. -Wl,-O1 -Wl,--as-needed -rdynamic -Wl,-export-dynamic'

Apr 25 '13 18:04 Olipro

I'd been looking for that! Thanks for the info @Olipro.

I assume this would also work well in conjunction with bundle config

Apr 25 '13 21:04 cinderblock

yep, bundle config build.charlock_holmes --with-ldflags='-L. -Wl,-O1 -Wl,--as-needed -rdynamic -Wl,-export-dynamic'

Apr 25 '13 22:04 Olipro

Thanks for the snippet. I think however the no-undefined is desired. We should instead manually add the missing libs, until this is fixed at least.

bundle config build.charlock_holmes --with-ldflags='-L. -Wl,-O1 -Wl,--as-needed -rdynamic -Wl,-export-dynamic -Wl,--no-undefined -lz -licuuc'

Apr 28 '13 21:04 cinderblock

@Olipro , thanks for this fix.It installs now on my gentoo machine ;)

May 25 '13 21:05 remoe

I still see this with charlock holmes 0.6.9.4:

Installing charlock_holmes (0.6.9.4) 
Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

    /usr/bin/ruby20 extconf.rb 
checking for main() in -licui18n... yes
checking for main() in -licui18n... yes
checking for unicode/ucnv.h... yes
  -- tar zxvf file-5.08.tar.gz
  -- ./configure --prefix=/sync1/gitlab/vendor/bundle/ruby/2.0.0/gems/charlock_holmes-0.6.9.4/ext/charlock_holmes/dst/ --disable-shared --enable-static --with-pic
  -- patch -p0 < ../file-soft-check.patch
  -- make -C src install
  -- make -C magic install
checking for main() in -lmagic_ext... yes
checking for magic.h... yes
creating Makefile

make
compiling encoding_detector.c
In file included from encoding_detector.c:3:0:
common.h:14:14: warning: 'charlock_new_enc_str' defined but not used [-Wunused-function]
compiling ext.c
In file included from ext.c:1:0:
common.h:14:14: warning: 'charlock_new_enc_str' defined but not used [-Wunused-function]
common.h:23:14: warning: 'charlock_new_str' defined but not used [-Wunused-function]
common.h:32:14: warning: 'charlock_new_str2' defined but not used [-Wunused-function]
compiling converter.c
In file included from converter.c:2:0:
common.h:23:14: warning: 'charlock_new_str' defined but not used [-Wunused-function]
common.h:32:14: warning: 'charlock_new_str2' defined but not used [-Wunused-function]
compiling transliterator.cpp
linking shared-object charlock_holmes/charlock_holmes.so
encoding_detector.o: In function `rb_get_supported_encodings':
encoding_detector.c:(.text+0xa0): undefined reference to `uenum_count'
encoding_detector.c:(.text+0xc1): undefined reference to `uenum_next'
encoding_detector.c:(.text+0x12a): undefined reference to `uenum_next'
encoding_detector.c:(.text+0x16c): undefined reference to `uenum_next'
encoding_detector.c:(.text+0x1aa): undefined reference to `uenum_next'
encoding_detector.c:(.text+0x1f1): undefined reference to `uenum_next'
encoding_detector.o:encoding_detector.c:(.text+0x22e): more undefined references to `uenum_next' follow
encoding_detector.o: In function `rb_encdec__alloc':
encoding_detector.c:(.text+0x73d): undefined reference to `u_errorName'
converter.o: In function `rb_converter_convert':
converter.c:(.text+0xb4): undefined reference to `ucnv_convert'
converter.c:(.text+0xf4): undefined reference to `ucnv_convert'
converter.c:(.text+0x18e): undefined reference to `u_errorName'
transliterator.o: In function `rb_transliterator_id_list':
transliterator.cpp:(.text+0x158): undefined reference to `u_errorName'
transliterator.o: In function `rb_transliterator_transliterate':
transliterator.cpp:(.text+0x2dd): undefined reference to `icu::UnicodeString::UnicodeString(char const*, int)'
transliterator.cpp:(.text+0x305): undefined reference to `icu::UnicodeString::~UnicodeString()'
transliterator.cpp:(.text+0x31b): undefined reference to `icu::UMemory::operator new(unsigned long)'
transliterator.cpp:(.text+0x331): undefined reference to `icu::UnicodeString::UnicodeString(char const*, int)'
transliterator.cpp:(.text+0x36a): undefined reference to `icu::UnicodeString::toUTF8(icu::ByteSink&) const'
transliterator.cpp:(.text+0x3b5): undefined reference to `icu::ByteSink::~ByteSink()'
transliterator.cpp:(.text+0x405): undefined reference to `u_errorName'
transliterator.cpp:(.text+0x46f): undefined reference to `icu::UMemory::operator delete(void*)'
transliterator.cpp:(.text+0x4a0): undefined reference to `icu::UnicodeString::~UnicodeString()'
transliterator.o: In function `icu::StringByteSink<std::string>::~StringByteSink()':
transliterator.cpp:(.text._ZN3icu14StringByteSinkISsED2Ev[_ZN3icu14StringByteSinkISsED5Ev]+0xf): undefined reference to `icu::ByteSink::~ByteSink()'
transliterator.o: In function `icu::StringByteSink<std::string>::~StringByteSink()':
transliterator.cpp:(.text._ZN3icu14StringByteSinkISsED0Ev[_ZN3icu14StringByteSinkISsED0Ev]+0x13): undefined reference to `icu::ByteSink::~ByteSink()'
transliterator.cpp:(.text._ZN3icu14StringByteSinkISsED0Ev[_ZN3icu14StringByteSinkISsED0Ev]+0x1c): undefined reference to `icu::UMemory::operator delete(void*)'
transliterator.o:(.data.rel.ro._ZTVN3icu14StringByteSinkISsEE[_ZTVN3icu14StringByteSinkISsEE]+0x28): undefined reference to `icu::ByteSink::GetAppendBuffer(int, int, char*, int, int*)'
transliterator.o:(.data.rel.ro._ZTVN3icu14StringByteSinkISsEE[_ZTVN3icu14StringByteSinkISsEE]+0x30): undefined reference to `icu::ByteSink::Flush()'
transliterator.o:(.data.rel.ro._ZTIN3icu14StringByteSinkISsEE[_ZTIN3icu14StringByteSinkISsEE]+0x10): undefined reference to `typeinfo for icu::ByteSink'
./libmagic_ext.a(compress.o): In function `uncompressgzipped':
/sync1/gitlab/vendor/bundle/ruby/2.0.0/gems/charlock_holmes-0.6.9.4/ext/charlock_holmes/src/file-5.08/src/compress.c:357: undefined reference to `inflateInit2_'
/sync1/gitlab/vendor/bundle/ruby/2.0.0/gems/charlock_holmes-0.6.9.4/ext/charlock_holmes/src/file-5.08/src/compress.c:363: undefined reference to `inflate'
/sync1/gitlab/vendor/bundle/ruby/2.0.0/gems/charlock_holmes-0.6.9.4/ext/charlock_holmes/src/file-5.08/src/compress.c:370: undefined reference to `inflateEnd'
collect2: error: ld returned 1 exit status
make: *** [charlock_holmes.so] Error 1


Gem files will remain installed in /sync1/gitlab/vendor/bundle/ruby/2.0.0/gems/charlock_holmes-0.6.9.4 for inspection.
Results logged to /sync1/gitlab/vendor/bundle/ruby/2.0.0/gems/charlock_holmes-0.6.9.4/ext/charlock_holmes/gem_make.out

An error occurred while installing charlock_holmes (0.6.9.4), and Bundler cannot continue.
Make sure that `gem install charlock_holmes -v '0.6.9.4'` succeeds before bundling.

Any idea why?

Nov 24 '13 12:11 vanthome

Have the same problem. It seems the build looks for static symbols (_ZTIN3icu14StringByteSinkISsEE) while the default version of ICU come as dynamic libraries (__imp___ZTIN3icu14StringByteSinkISsEE). I tried to compile ICU as static libraries but then I got another errors compiling charlock_holmes native extensions, including "bad realloc address" stuff in the static libs... I am on cygwin, does this issue reproduce on Linux distros?

Nov 27 '13 03:11 laurentpsychedelic

It builds OK on Debian, so I guess this is particular to builds on Cygwin? @vanthome what is your platform?

Nov 27 '13 04:11 laurentpsychedelic

Gentoo

Nov 27 '13 05:11 vanthome

It seems that the gem has not been updated yet on rubygems.org, so it will continue to fail when installed from there without some manual patching. For Cygwin folks who may be interested, it seems that ICU v.5.2 (required to build this gem) is not yet available as a Cygwin package (the last available version is v.4.8), so installation from source is needed in order to build this gem on Cygwin.

Nov 28 '13 04:11 laurentpsychedelic

@laurentpsychedelic where are you getting this information about the ICU requirements?

Dec 09 '13 05:12 ghost

@svnpenn I deduced that from the symbols in the error message (link error of GCC). The symbols seem to include the version number (something like _icu48... or _icu52...). However I may have misunderstood the situation, so sorry if it's wrong or misleading. It may also be due to Cygwin package. The only sure thing is that you cannot install the charlock_holmes gem using the packaged ICU (which happens to be 4.8) coming with Cygwin, so I had to compile ICU from source (which was 5.2). I must admit that I did not take the time to analyze the error in details as I managed to obtain a working gem. If I find time I will try again the installation from zero and report my findings as soon as possible.

Dec 10 '13 07:12 laurentpsychedelic

I am on Cygwin and had to do the following to "bundle install" github/linguist, which requires charlock_holmes 0.6.9.4:

Install these packages

libicu-devel (4.8.1-1) (thanks, http://stackoverflow.com/q/15553792) patch (thanks, https://github.com/brianmario/charlock_holmes/issues/44)
Run these commands (thanks, folks participating in this issue--I only needed to add "-licuuc", even the z library wasn't needed)

bundle config build.charlock_holmes --with-dldflags='-licuuc -Wl,--export-all-symbols -Wl,--enable-auto-image-base,--enable-auto-import' bundle install

Success!

Dec 24 '13 04:12 m5n

This did the trick for me on Ruby 1.9.3: bundle config build.charlock_holmes --with-ldflags='-L. -Wl,-O1 -Wl,--as-needed -rdynamic -Wl,-export-dynamic -Wl,--no-undefined -lz -licuuc'

I still see the problem on Ruby 2.0 though. Anyone else encountered this on Ruby 2.0?

Jan 08 '14 09:01 radamanthus

Yes, I've seen it on Ruby 2.0 as well. And I did the same bundle config that you have done, at it "solved" it for me as well.

Jan 08 '14 10:01 elestedt

For anyone having troubles deploying code to a server and are being affected by this issue what you want to do is run the above mentioned bundle config changes locally within the project and then deploy the changes.

Use the following commands from the terminal in your development environment.

$ mkdir .bundle
$ bundle config --local build.charlock_holmes --with-ldflags='-L. -Wl,-O1 -Wl,--as-needed -rdynamic -Wl,-export-dynamic -Wl,--no-undefined -lz -licuuc'

This will locally create a file (./bundle/config). Then commit this file into version control and deploy these changes to your server. SORTED!

(Thanks to EngineYard tech support for helping me out towards this solution)

Feb 18 '14 08:02 evolve2k