SSL connection issues with Brig on Debian

[sadoyan]

When I run brig server I get brig: ProtocolError "error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib" is any HTTPS endpoint is configured in brig.yaml for example I get error on this sqsEndpoint: https://sqs.us-east-1.amazonaws.com/ , but when I run brig with sqsEndpoint: http://sqs.us-east-1.amazonaws.com/ it does not throws error. I have tried fake SQS servers as well, if it is under SSL I get error. I have compiled brig strongly following README on Debian 9 64 bit

[sadoyan]

this does not happens on Ubuntu 16.04, do you have some special requirements ?

[fisx]

Thanks for reporting this.

We have seen this error, but have not had time to investigate. Do you have the option of running wire-server on Ubuntu?

If not, and if you have time to find out more, you can reproduce this by running make integration in services/brig. Any help with this would be greatly appreciated!

[sadoyan]

Now I run wire-server component on Ubuntu 16.0.4 seems without any visible issue. At least I can add user and login to swagger-ui. It looks like this is Debian's OpenSSL version issue.

[fisx]

I stand corrected by @tiago-loureiro (-:

We have taken the time to investigate, and we found that the problem occurs if you have different versions of either openssl or libssl installed. It has been reproduced on a system with the following:

• openssl-1.1.0f-3+deb9u2
• libssl1.0.0:amd64-1.0.1t-1+deb8u6
• libssl1.0.2:amd64-1.0.2l-2+deb9u3
• libssl1.1:amd64-1.1.0f-3+deb9u2

So removing an appropriate subset of these may help, but it may not be an option due to your dependency tree.

I was correct in that any help on this is greatly appreciated. (-:

[fisx]

Now I run wire-server component on Ubuntu 16.0.4 seems without any visible issue.

Excellent.

Please leave the issue open in any case for people who want to use Debian.

[sadoyan]

yes you are right seems default package installation of Debian 9 have different versions of OpenSSL and libssl. Indeed strange thing version of libssl comes as dependency for some packages in Debian 9. Debian 9 libssl1.0.2:amd64 1.0.2l-2+deb9u3 libssl1.1:amd64 1.1.0f-3+deb9u2 openssl 1.1.0f-3+deb9u2 Ubuntu 16.04 libssl1.0.0:amd64 1.0.2g-1ubuntu4.12 openssl 1.0.2g-1ubuntu4.12

[jschaul]

Additional info on this one: This issue has come up for us only on debian distributions (both on jessie and on stretch). Other distributions (fedora, arch, ubuntu, macOS) do not have this problem. Only the brig service is affected. If anyone manages to figure out what causes this problem on debian, or how to fix it, please let us know.

[julialongtin]

Per https://github.com/wireapp/wire-server/issues/430, we believe this issue, and the one described in 430 are closely related, and possibly the same issue.

To reproduce the exact issue shown in #430 :

# start up the docker services
$ ./deploy/docker-ephemeral/run.sh

Once the services have started (you will see the docker-ephemeral_db_migration scripts exit), open up another terminal, and:

$ ./services/start-services-only.sh

now open up another terminal window, and try to activate an email address:

$ curl 127.0.0.1:8082/activate/send -X POST -H 'Content-Type: application/json' -d '{"email":"[email protected]"}'

The response from curl will be 500, internal server error.

In the terminal window that you ran start-services-only, you will see the following errors:

[brig] W, logger=cassandra.brig, exception=user error (error:00000000:lib(0):func(0):reason(0))
[brig] E, request=N/A, user error (error:00000000:lib(0):func(0):reason(0))

[julialongtin]

This is the same error message as https://github.com/openssl/openssl/issues/7488 . Examining our exception handling.

[fisx]

the exception is thrown synchronously here: if you error out before this line (after mkActivationKey has returned), it throws; after that line the puzzling ssl-non-ssl-error is thrown in stead.

[fisx]

see also: https://github.com/wireapp/wire-server/pull/661

we may want to try to reproduce this on that version of cql-io.

[julialongtin]

Note that when we've added code to catch the exception, we're finding it has type 'IOException'.

[julialongtin]

This exception is being thrown when we call runClient.

[julialongtin]

in sendActivationCode the offending operation is performed when we call Data.lookupActivationCode.

[julialongtin]

and in lookupActivationCode, the problem occurs when we call query1, which is what calls runClient.

[jonassmedegaard]

yes you are right seems default package installation of Debian 9 have different versions of OpenSSL and libssl. Indeed strange thing version of libssl comes as dependency for some packages in Debian 9. Debian 9 libssl1.0.2:amd64 1.0.2l-2+deb9u3 libssl1.1:amd64 1.1.0f-3+deb9u2 openssl 1.1.0f-3+deb9u2 Ubuntu 16.04 libssl1.0.0:amd64 1.0.2g-1ubuntu4.12 openssl 1.0.2g-1ubuntu4.12

Debian developer here. When needing to link with OpenSSL 1.0.x install package libssl1.0-dev, and if needing to link with OpenSSL 1.1.x install libssl-dev. Those package conflict with each other.

Linking with OpenSSL 1.0.x is deprecated and only available in Debian oldstable: https://packages.debian.org/search?keywords=libssl1.0-dev

Debian has seen updates to libssl/openssl and is now at versions 1.0.2u (only on oldstable) and varying (all newer than quoted above) for OpenSSL 1.1.x: https://packages.debian.org/search?keywords=libssl-dev

All that said, I therefore wonder: Is this issue still repeatable, with up-to-date packages for each Debian release, and when following the advice of installing the appropriate -dev package when linking?

[jonassmedegaard]

If/when this issue is solved, please remember to update https://github.com/wireapp/wire-server/blob/develop/docs/developer/dependencies.md

Stumbling across that note was what triggered my comments here: Obviously as proud Debian developer I cannot rest with such discouragement note in high-quality projects like this one :-)