How to build nginx with openssl from qat-engine?

[kkurzacz-intel]

Problem

I can't find a way to correctly build nginx with the qat-enabled openssl, which does not stay in common system path

Reason why problem happenned

I want to compare async nginx with qat support vs common nginx from linux repository. To compare, I also want openssl with qat support to not override the stock system version. In another words I want to compare all stock (everything from official repo) nginx vs nginx with qat enabled and all it's dependent libraries.

What I have did

I started with build of qat engine. I have chosen "Example 5: Using make depend for building dependant libraries and build QAT Engine" from QAT Engine example builds. This will also take care about installing openssl. So I ./configure qat engine with following flag:

--with-openssl_install_dir=/usr/local/ssl

Now when I try to ./configure and make async nginx, everything is fine until I try to use nginx.QAT-sample.conf file. Then I'm getting following errors:

$ sudo ./objs/nginx -p /home/kkurzacz/benchmark/asynch_mode_nginx
nginx: [emerg] QAT Engine failed: ENABLE_SW_FALLBACK
nginx: [emerg] ssl engine send ctrl failed
nginx: [emerg] ssl engine set failed

FYI: As I mentioned, I want to keep nginx from linux repo as well, so that's why I don't install this async nginx. Probably I should use the prefix flag to install it somewhere, but let's postpone it for now.

Reading issue #47 made me verify openssl, and it seems all right:

/usr/local/ssl/bin$ ./openssl engine -t -c -vvvv qatengine
(qatengine) Reference implementation of QAT crypto engine(qat_hw & qat_sw) v1.4.0
 [RSA, AES-128-CBC-HMAC-SHA256, AES-256-CBC-HMAC-SHA256, ChaCha20-Poly1305, id-aes128-GCM, id-aes192-GCM, id-aes256-GCM, SHA3-256, SHA3-384, SHA3-512, TLS1-PRF, X25519, X448, SM2]
     [ available ]                                                                                                                                    ENABLE_EXTERNAL_POLLING: Enables the external polling interface to the engine.
          (input flags): NO_INPUT

My ./configure looks like follows:

$ ./configure
--with-http_ssl_module \
--add-dynamic-module=modules/nginx_qatzip_module \
--add-dynamic-module=modules/nginx_qat_module/ \
--with-cc-opt="-DNGX_SECURE_MEM -I/usr/local/ssl/include -I/home/kkurzacz/benchmark/qat/qatzip/include -Wno-error=deprecated-declarations" \
--with-ld-opt="-Wl,-rpath=/usr/local/ssl/lib64 -L/usr/local/ssl/lib64 -L/home/kkurzacz/benchmark/qat/qatzip/src -lqatzip -lz"

So despite pointing to compiled openssl libs, I cannot start nginx.

I have found the --with-openssl flag for ./configure. Initially I thought I can use to point to binary, so I tried --with-openssl=/usr/local/ssl. However, this flag seems to build search path using .openssl subfolder. When I figured out that this flag is meant to point to openssl source files, I tried to use path of the openssl downloaded during make depend of qat engine. It downloads openssl to openssl subfolder of main repository.

So, qat engine repository is /home/kkurzacz/benchmark/qat/qat-engine Openssl is then downloaded into /home/kkurzacz/benchmark/qat/qat-engine/openssl

However, it also doesn't work (this way of configure of async nginx):

./configure 
# ... \
--with-openssl=/home/kkurzacz/benchmark/qat/qat-engine/openssl

$ make
make -f objs/Makefile
make[1]: Entering directory '/home/kkurzacz/benchmark/asynch_mode_nginx'
cd /home/kkurzacz/benchmark/qat/qat-engine/openssl \
&& if [ -f Makefile ]; then make clean; fi \
&& ./config --prefix=/home/kkurzacz/benchmark/qat/qat-engine/openssl/.openssl no-shared no-threads  \
&& make \
&& make install_sw LIBDIR=lib
make[2]: Entering directory '/home/kkurzacz/benchmark/qat/qat-engine/openssl'
rm -f libcrypto.so.3
rm -f libcrypto.so
rm -f libssl.so.3
rm -f libssl.so
rm -f apps/libapps.a libcrypto.a libssl.a providers/libcommon.a providers/libdefault.a providers/liblegacy.a test/libtestutil.a
rm -f *.ld
...
make[2]: *** [Makefile:3363: clean] Error 1
make[2]: Leaving directory '/home/kkurzacz/benchmark/qat/qat-engine/openssl'
make[1]: *** [objs/Makefile:1310: /home/kkurzacz/benchmark/qat/qat-engine/openssl/.openssl/include/openssl/ssl.h] Error 2
make[1]: Leaving directory '/home/kkurzacz/benchmark/asynch_mode_nginx'
make: *** [Makefile:10: build] Error 2

What is mostly confusing for me, is that search path for openssl libs is done like:

user-provided-path + .openssl + relative-paths
(/home/kkurzacz/benchmark/qat/qat-engine/openssl) + (/.openssl) + (/include/openssl/ssl.h)

Can you please tell me how to build async nginx with my openssl lib which is downloaded via make depend of qat engine?

[Yogaraj-Alamenda]

@kkurzacz-intel Apologies for the late reply. Is the issue related to https://github.com/intel/asynch_mode_nginx/issues/72?

[Yogaraj-Alamenda]

The issue mentioned here is fixed with the commit below in QAT Engine and released in QAT Engine v1.6.0 https://github.com/intel/QAT_Engine/commit/3a1fca3138c96054721bebe19861b0cd6dc449af. Hence closing this.