Installing fails on Debian Stable (buster) with "Cannot locate symbol 'sk_num' in native library 'libssl.so'"

[zerodogg]

I've installed libssl-dev which provides libssl.so. Without libssl-dev installed I get Cannot locate native library 'libssl.so': libssl.so: cannot open shared object file: No such file or directory instead.

Here's trying to install the module on Debian Stable with rakudo v2020.12.

$ zef install OpenSSL
===> Searching for: OpenSSL
===> Testing: OpenSSL:ver<0.1.23>:auth<github:sergot>
[OpenSSL] # NETWORK_TESTING was not set
[OpenSSL] Cannot locate symbol 'sk_num' in native library 'libssl.so'
[OpenSSL]   in method setup at /opt/rakudo-pkg/share/perl6/core/sources/947BDAB9F96E0E5FCCB383124F923A6BF6F8D76B (NativeCall) line 298
[OpenSSL]   in sub sk_num at /opt/rakudo-pkg/share/perl6/core/sources/947BDAB9F96E0E5FCCB383124F923A6BF6F8D76B (NativeCall) line 587
[OpenSSL]   in block <unit> at t/10-client-ca-file.t line 10
[OpenSSL] # You planned 7 tests, but ran 1
[OpenSSL] # NETWORK_TESTING was not set
===> Testing [FAIL]: OpenSSL:ver<0.1.23>:auth<github:sergot>
Aborting due to test failure: OpenSSL:ver<0.1.23>:auth<github:sergot> (use --force-test to override

[lizmat]

I just ran into this as well :-(

[lizmat]

Oddly enough, cloning the OpenSSL repo into a local dir, all tests pass. And thus zef install . works like a charm.

[raydiak]

I am also encountering this problem on Kubuntu 20.04, where libssl.so is a symlink to libssl.so.1.1

[usev6]

I've hit the same problem on Debian 10 (buster) that comes with openssl (and libssl) 1.1.1d.

I've used rakudo-pkg for this installation.

$ raku --version
Welcome to Rakudo(tm) v2021.05.
Implementing the Raku(tm) programming language v6.d.
Built on MoarVM version 2021.05.

If I'm not mistaken things work smoothly if libssl-dev is installed before trying to run zef install OpenSSL the first time:

$ sudo apt install libssl-dev
[...]
libssl-dev is already the newest version (1.1.1d-0+deb10u6).
[...]
$ /opt/rakudo-pkg/bin/install-zef
[...]
$ PATH=~/.raku/bin:$PATH
$ zef --version
v0.11.7
$ zef install OpenSSL
===> Searching for: OpenSSL
===> Updating fez mirror: http://360.zef.pm/
===> Updating cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json
===> Updating p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json
===> Updated p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json
===> Updated fez mirror: http://360.zef.pm/
===> Updated cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json
===> Building: OpenSSL:ver<0.1.24>
===> Building [OK] for OpenSSL:ver<0.1.24>
===> Testing: OpenSSL:ver<0.1.24>
[OpenSSL] # NETWORK_TESTING was not set
[OpenSSL] # NETWORK_TESTING was not set
===> Testing [OK] for OpenSSL:ver<0.1.24>
===> Installing: OpenSSL:ver<0.1.24>
$ raku -e 'use OpenSSL; say OpenSSL.new'
OpenSSL.new(ctx => OpenSSL::Ctx::SSL_CTX.new(method => OpenSSL::Method::SSL_METHOD.new(version => 65536)), ssl => OpenSSL::SSL::SSL.new(version => 772, type => 0, method => OpenSSL::Method::SSL_METHOD.new(version => 65536), rbio => OpenSSL::Bio::BIO, wbio => OpenSSL::Bio::BIO, bbio => OpenSSL::Bio::BIO, rwstate => 1, in_handshake => 0, handshake_func => NativeCall::Types::Pointer, server => 1, new_session => 0, quiet_shutdown => 0, shutdown => 0, state => 0, rstate => 0), client => Bool::False, using-bio => Bool::False, bio-read-buf => Buf[uint8].new(), net-write => Any, net-read => Any, net-bio => Any, internal-bio => Any)

Now I nuke the newly installed stuff, remove libssl-dev and start anew. As expected the installation fails with error messages pointing to the missing libssl.so:

$ rm -rf ~/.raku/ ~/.zef/
$ sudo apt -y remove libssl-dev
[...]
Removing libssl-dev:amd64 (1.1.1d-0+deb10u6) ...
$ /opt/rakudo-pkg/bin/install-zef
[...]
$ zef --version
v0.11.7
$ zef install OpenSSL
[...]
===> Testing: OpenSSL:ver<0.1.24>
[OpenSSL] Cannot locate native library 'libssl.so': libssl.so: cannot open shared object file: No such file or directory
[OpenSSL]   in method setup at /opt/rakudo-pkg/share/perl6/core/sources/947BDAB9F96E0E5FCCB383124F923A6BF6F8D76B (NativeCall) line 299
[OpenSSL]   in sub EVP_aes_128_cbc at /opt/rakudo-pkg/share/perl6/core/sources/947BDAB9F96E0E5FCCB383124F923A6BF6F8D76B (NativeCall) line 589
[OpenSSL]   in method new at /home/christian/.zef/store/openssl.git/06ffd7c48970e52d6e261ea3e13601166c7b1364/lib/OpenSSL.pm6 (OpenSSL) line 36
[OpenSSL]   in block <unit> at t/01-basic.t line 6
[...]
===> Testing [FAIL]: OpenSSL:ver<0.1.24>
Aborting due to test failure: OpenSSL:ver<0.1.24> (use --force-test to override)

I'd expect that (re-)installing libssl-dev would fix the problem, but then the complaint about not finding sk_num pops up:

$ sudo apt install libssl-dev
[...]
$ zef install OpenSSL
===> Searching for: OpenSSL
===> Building: OpenSSL:ver<0.1.24>
===> Building [OK] for OpenSSL:ver<0.1.24>
===> Testing: OpenSSL:ver<0.1.24>
[OpenSSL] # NETWORK_TESTING was not set
[OpenSSL] Cannot locate symbol 'sk_num' in native library 'libssl.so'
[OpenSSL]   in method setup at /opt/rakudo-pkg/share/perl6/core/sources/947BDAB9F96E0E5FCCB383124F923A6BF6F8D76B (NativeCall) line 299
[OpenSSL]   in sub sk_num at /opt/rakudo-pkg/share/perl6/core/sources/947BDAB9F96E0E5FCCB383124F923A6BF6F8D76B (NativeCall) line 589
[OpenSSL]   in block <unit> at t/10-client-ca-file.t line 10
[OpenSSL] # You planned 7 tests, but ran 1
[OpenSSL] # NETWORK_TESTING was not set
===> Testing [FAIL]: OpenSSL:ver<0.1.24>
Aborting due to test failure: OpenSSL:ver<0.1.24> (use --force-test to override)

After nuking ~/.raku and ~/.zef the installation works again.

$ rm -rf ~/.raku/ ~/.zef/
$ /opt/rakudo-pkg/bin/install-zef
[...]
$ zef install OpenSSL
===> Searching for: OpenSSL
===> Updating fez mirror: http://360.zef.pm/
===> Updating cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json
===> Updating p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json
===> Updated p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json
===> Updated cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json
===> Updated fez mirror: http://360.zef.pm/
===> Building: OpenSSL:ver<0.1.24>
===> Building [OK] for OpenSSL:ver<0.1.24>
===> Testing: OpenSSL:ver<0.1.24>
[OpenSSL] # NETWORK_TESTING was not set
[OpenSSL] # NETWORK_TESTING was not set
===> Testing [OK] for OpenSSL:ver<0.1.24>
===> Installing: OpenSSL:ver<0.1.24>
$ raku -e 'use OpenSSL; say OpenSSL.new'
OpenSSL.new(ctx => OpenSSL::Ctx::SSL_CTX.new(method => OpenSSL::Method::SSL_METHOD.new(version => 65536)), ssl => OpenSSL::SSL::SSL.new(version => 772, type => 0, method => OpenSSL::Method::SSL_METHOD.new(version => 65536), rbio => OpenSSL::Bio::BIO, wbio => OpenSSL::Bio::BIO, bbio => OpenSSL::Bio::BIO, rwstate => 1, in_handshake => 0, handshake_func => NativeCall::Types::Pointer, server => 1, new_session => 0, quiet_shutdown => 0, shutdown => 0, state => 0, rstate => 0), client => Bool::False, using-bio => Bool::False, bio-read-buf => Buf[uint8].new(), net-write => Any, net-read => Any, net-bio => Any, internal-bio => Any)

So I suspect something is left over from the first (unsuccessful) try. Does that ring a bell for someone?

It would be nice if someone else could verify that things work as expected if libssl-dev is available when trying to run zef install OpenSSL for the first time.

[ugexe]

@usev6 Based on your observations I suspect you could reproduce the issue with a local repo (i.e. using zef install . instead of zef install OpenSSL) and then "fix" it by deleting .precomp/ (instead of deleting ~/.zef)

[usev6]

That was spot on:

$ git clone https://github.com/sergot/openssl.git
$ cd openssl
$ sudo apt -y remove libssl-dev
$ zef install .
[...]
[OpenSSL] Cannot locate native library 'libssl.so': libssl.so: cannot open shared object file: No such file or directory
[...]
===> Testing [FAIL]: OpenSSL:ver<0.1.24>
Aborting due to test failure: OpenSSL:ver<0.1.24> (use --force-test to override)
$ sudo apt install libssl-dev
$ zef install .
[...]
[OpenSSL] Cannot locate symbol 'sk_num' in native library 'libssl.so'
[...]
===> Testing [FAIL]: OpenSSL:ver<0.1.24>
Aborting due to test failure: OpenSSL:ver<0.1.24> (use --force-test to override)
$ rm -r .precomp
$ zef install .
===> Building: OpenSSL:ver<0.1.24>
===> Building [OK] for OpenSSL:ver<0.1.24>
===> Testing: OpenSSL:ver<0.1.24>
[OpenSSL] # NETWORK_TESTING was not set
[OpenSSL] # NETWORK_TESTING was not set
===> Testing [OK] for OpenSSL:ver<0.1.24>
===> Installing: OpenSSL:ver<0.1.24>

[usev6]

Unfortunately the new version 0.1.25 (with the patch from https://github.com/sergot/openssl/commit/72ce2495b6) fails to install for me even with libssl-dev available:

$ sudo apt -y install libssl-dev
[...]
$ git clone https://github.com/sergot/openssl.git
[...]
$ cd openssl
$ zef install .
===> Searching for missing dependencies: PathTools, JSON::Fast
===> Updating fez mirror: http://360.zef.pm/
===> Updating cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json
===> Updating p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json
===> Updated p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json
===> Updated fez mirror: http://360.zef.pm/
===> Updated cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json
===> Building: OpenSSL:ver<0.1.25>
===> Building [OK] for OpenSSL:ver<0.1.25>
===> Testing: PathTools:ver<0.1.1>:auth<github:ugexe>
===> Testing [OK] for PathTools:ver<0.1.1>:auth<github:ugexe>
===> Testing: JSON::Fast:ver<0.16>
===> Testing [OK] for JSON::Fast:ver<0.16>
===> Testing: OpenSSL:ver<0.1.25>
[OpenSSL] # NETWORK_TESTING was not set
[OpenSSL] Unable to read key
[OpenSSL]   in method new at /home/christian/openssl/lib/OpenSSL/RSATools.pm6 (OpenSSL::RSATools) line 23
[OpenSSL]   in block <unit> at t/03-rsa.t line 10
[OpenSSL] # NETWORK_TESTING was not set
===> Testing [FAIL]: OpenSSL:ver<0.1.25>
Aborting due to test failure: OpenSSL:ver<0.1.25> (use --force-test to override)

[usev6]

I don't know if this is helpful, but adding a no precompilation at the top of lib/OpenSSL/Stack.pm6 seems to make the original problem go away.

My next thought was: That's not a general solution. But then I started to wonder, how an update of a native library would (and should) be handled. I don't understand what precompilation does for subs that are defined with is native. (Maybe that's not even relevant here and the problem is that the precompiled version of sub real_symbol returns the wrong result in https://github.com/sergot/openssl/blob/67269bc82e/lib/OpenSSL/Stack.pm6#L15-L19.)

[ugexe]

If installation was working before and after manually deleting the .precomp directory then I wonder if https://github.com/sergot/openssl/commit/72ce2495b6 is actually deleting all the precomp files

[usev6]

This is weird. It looks like the new problem (Unable to read key in OpenSSL::RSATools) goes away randomly.

I just tried to run zef install . a couple of times and it succeeded at the fifth attempt:

$ sudo apt -y install libssl-dev
[...]
$ rm -rf ~/.raku ~/.zef ~/openssl
$ /opt/rakudo-pkg/bin/install-zef
[...]
$ export PATH=~/.raku/bin:$PATH
$ git clone https://github.com/sergot/openssl.git
$ cd ~/openssl
$ zef install .
[...]
[OpenSSL] Unable to read key
[OpenSSL]   in method new at /home/christian/openssl/lib/OpenSSL/RSATools.pm6 (OpenSSL::RSATools) line 23
[OpenSSL]   in block <unit> at t/03-rsa.t line 10
[...]
Aborting due to test failure: OpenSSL:ver<0.1.25> (use --force-test to override)
$ zef install .
[...]
[OpenSSL] Unable to read key
[OpenSSL]   in method new at /home/christian/openssl/lib/OpenSSL/RSATools.pm6 (OpenSSL::RSATools) line 23
[OpenSSL]   in block <unit> at t/03-rsa.t line 10
[...]
Aborting due to test failure: OpenSSL:ver<0.1.25> (use --force-test to override)
$ zef install .
[...]
[OpenSSL] Unable to read key
[OpenSSL]   in method new at /home/christian/openssl/lib/OpenSSL/RSATools.pm6 (OpenSSL::RSATools) line 23
[OpenSSL]   in block <unit> at t/03-rsa.t line 10
[...]
Aborting due to test failure: OpenSSL:ver<0.1.25> (use --force-test to override)
$ zef install .
[...]
[OpenSSL] Unable to read key
[OpenSSL]   in method new at /home/christian/openssl/lib/OpenSSL/RSATools.pm6 (OpenSSL::RSATools) line 23
[OpenSSL]   in block <unit> at t/03-rsa.t line 10
[...]
Aborting due to test failure: OpenSSL:ver<0.1.25> (use --force-test to override)
$ zef install .
[...]
===> Testing [OK] for OpenSSL:ver<0.1.25>
[...]
===> Installing: OpenSSL:ver<0.1.25>

[usev6]

I've opened a separate issue for the new problem: https://github.com/sergot/openssl/issues/87

[usev6]

With https://github.com/sergot/openssl/issues/87 out of the way, the installation of OpenSSL (0.1.26) behaves as expected:

1. fails with missing package libssl-dev
2. but succeeds after installing that package

$ rm -rf ~/.zef/ ~/.raku/  ## cleanup old installation
$ sudo apt -y remove libssl-dev  ## remove needed package to see expected error
[...]
$ /opt/rakudo-pkg/bin/install-zef
[...]
$ export PATH=~/.raku/bin:$PATH
$ zef install OpenSSL
[...]
[OpenSSL] Cannot locate native library 'libssl.so': libssl.so: cannot open shared object file: No such file or directory
[...]
===> Testing [FAIL]: OpenSSL:ver<0.1.26>
Aborting due to test failure: OpenSSL:ver<0.1.26> (use --force-test to override)
$ sudo apt -y install libssl-dev  ## provide needed package
$ zef install OpenSSL
[...]
===> Installing: OpenSSL:ver<0.1.26>
$ raku -e 'use OpenSSL; say "alive"'
alive

To be sure, I've tried the above commands a couple of times, but the results were consistent.

Maybe someone else could double check and then this issue could be closed?

[raydiak]

I've switched OSes since I last tried this, but I can confirm that this module now installs successfully without errors on Manjaro which currently provides OpenSSL 1.1.1k