Memory leak

[lathakris]

Hi,

I am using the rest client library to make an SSL connection to a HTTP server and make REST calls. I am using the release with the pthread fix. (1.0.2). I initiate the SSL connection and then keep doing the REST calls over the same connection until it returns error in which case I reconnect.

I am seeing that there seems to be a memory leak, the program is eventually running out of memory. I typically create the request, add the RestFilter_execute_curl_request in the filter chain and do RestClient_execute_request. At the end of the function, I free the filter, destroy the request and response.

I also set RestClient_add_curl_config_handler(&rcHandle, rest_disable_ssl_cert_check); to trust the cert. The RestClient handle I keep reusing all the time.

Could you please let me know what went wrong here. Am I missing any other cleanup ? Thanks in advance for your response.

I used valgrind to debug and here is the sample of the trace. (there is more trace if you need)

==1041== 8 bytes in 1 blocks are still reachable in loss record 1 of 637 ==1041== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==1041== by 0x5727DB2: CRYPTO_malloc (in /lib/libcrypto.so.1.0.0) ==1041== by 0x57A4D62: BUF_strndup (in /lib/libcrypto.so.1.0.0) ==1041== by 0x57FA5A6: ??? (in /lib/libcrypto.so.1.0.0) ==1041== by 0x57FAC63: CONF_module_add (in /lib/libcrypto.so.1.0.0) ==1041== by 0x57FAFED: OPENSSL_load_builtin_modules (in /lib/libcrypto.so.1.0.0) ==1041== by 0x54B3388: Curl_ossl_init (in /usr/lib/libcurl.so.4) ==1041== by 0x549BEC4: global_init (in /usr/lib/libcurl.so.4) ==1041== by 0x549C157: curl_easy_init (in /usr/lib/libcurl.so.4) ==1041== by 0x438852: RestFilter_execute_curl_request (rest_client.c:545) ==1041== by 0x409712: doAuth (restapi.c:191) ==1041== by 0x408172: checkFile (main.c:73) ==1041== ==1041== 8 bytes in 1 blocks are still reachable in loss record 2 of 637 ==1041== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==1041== by 0x5727DB2: CRYPTO_malloc (in /lib/libcrypto.so.1.0.0) ==1041== by 0x579EEA3: ??? (in /lib/libcrypto.so.1.0.0) ==1041== by 0x579F478: ENGINE_add (in /lib/libcrypto.so.1.0.0) ==1041== by 0x57A47D0: ENGINE_load_rsax (in /lib/libcrypto.so.1.0.0) ==1041== by 0x57A12DD: ENGINE_load_builtin_engines (in /lib/libcrypto.so.1.0.0) ==1041== by 0x54B338D: Curl_ossl_init (in /usr/lib/libcurl.so.4) ==1041== by 0x549BEC4: global_init (in /usr/lib/libcurl.so.4) ==1041== by 0x549C157: curl_easy_init (in /usr/lib/libcurl.so.4) ==1041== by 0x438852: RestFilter_execute_curl_request (rest_client.c:545) ==1041== by 0x409712: doAuth (restapi.c:191) ==1041== by 0x408172: checkFile (main.c:73)

[jasoncwik]

Looks like the curl handle might not be getting freed. Can you write a small program you can share that reproduces the issue?

Sent from my iPhone

On Aug 26, 2016, at 4:50 PM, Latha Krishnamurthi [email protected] wrote:

Hi,

I am using the rest client library to make an SSL connection to a HTTP server and make REST calls. I am using the release with the pthread fix. (1.0.2). I initiate the SSL connection and then keep doing the REST calls over the same connection until it returns error in which case I reconnect.

I am seeing that there seems to be a memory leak, the program is eventually running out of memory. I typically create the request, add the RestFilter_execute_curl_request in the filter chain and do RestClient_execute_request. At the end of the function, I free the filter, destroy the request and response.

I also set RestClient_add_curl_config_handler(&rcHandle, rest_disable_ssl_cert_check); to trust the cert. The RestClient handle I keep reusing all the time.

Could you please let me know what went wrong here. Am I missing any other cleanup ? Thanks in advance for your response.

I used valgrind to debug and here is the sample of the trace. (there is more trace if you need)

==1041== 8 bytes in 1 blocks are still reachable in loss record 1 of 637 ==1041== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==1041== by 0x5727DB2: CRYPTO_malloc (in /lib/libcrypto.so.1.0.0) ==1041== by 0x57A4D62: BUF_strndup (in /lib/libcrypto.so.1.0.0) ==1041== by 0x57FA5A6: ??? (in /lib/libcrypto.so.1.0.0) ==1041== by 0x57FAC63: CONF_module_add (in /lib/libcrypto.so.1.0.0) ==1041== by 0x57FAFED: OPENSSL_load_builtin_modules (in /lib/libcrypto.so.1.0.0) ==1041== by 0x54B3388: Curl_ossl_init (in /usr/lib/libcurl.so.4) ==1041== by 0x549BEC4: global_init (in /usr/lib/libcurl.so.4) ==1041== by 0x549C157: curl_easy_init (in /usr/lib/libcurl.so.4) ==1041== by 0x438852: RestFilter_execute_curl_request (rest_client.c:545) ==1041== by 0x409712: doAuth (restapi.c:191) ==1041== by 0x408172: checkFile (main.c:73) ==1041== ==1041== 8 bytes in 1 blocks are still reachable in loss record 2 of 637 ==1041== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==1041== by 0x5727DB2: CRYPTO_malloc (in /lib/libcrypto.so.1.0.0) ==1041== by 0x579EEA3: ??? (in /lib/libcrypto.so.1.0.0) ==1041== by 0x579F478: ENGINE_add (in /lib/libcrypto.so.1.0.0) ==1041== by 0x57A47D0: ENGINE_load_rsax (in /lib/libcrypto.so.1.0.0) ==1041== by 0x57A12DD: ENGINE_load_builtin_engines (in /lib/libcrypto.so.1.0.0) ==1041== by 0x54B338D: Curl_ossl_init (in /usr/lib/libcurl.so.4) ==1041== by 0x549BEC4: global_init (in /usr/lib/libcurl.so.4) ==1041== by 0x549C157: curl_easy_init (in /usr/lib/libcurl.so.4) ==1041== by 0x438852: RestFilter_execute_curl_request (rest_client.c:545) ==1041== by 0x409712: doAuth (restapi.c:191) ==1041== by 0x408172: checkFile (main.c:73)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[lathakris]

Hi Jason, thanks for your quick response. I have picked out the selected API and written the test program. Please let me know if you need more info. This is a HTTPS connection to a web server. It is a token based authentication request. Further queries are done over the established HTTPS connection.

test.txt

[lathakris]

if we just do this in a program, it leaks memory... ==========> CURL *curl = curl_easy_init(); curl_easy_cleanup(curl);

or even this ======> curl_global_init(CURL_GLOBAL_ALL); CURL *curl = curl_easy_init(); curl_easy_cleanup(curl); curl_global_cleanup();

seems like some open ssl leak ? Have you guys encountered this problem ? I repeatedly keep doing this rest execute request as and when the requests come.. so the leak is piling up.

Thanks,

[lathakris]

curl version we are using and corresponding openssl.

curl --version

curl 7.47.1 (x86_64-pc-linux-gnu) libcurl/7.47.1 OpenSSL/1.0.1f zlib/1.2.8 Protocols: dict file ftp ftps http https Features: AsynchDNS IPv6 Largefile NTLM SSL libz TLS-SRP UnixSockets

[jasoncwik]

I think I know what the issue is... Before using this library, your application needs to call:

        curl_global_init(CURL_GLOBAL_DEFAULT);

If you don't, global_init() will get called for every request implicitly by curl_easy_init() and causes the memory leak. Once I did this to the test harness, valgrind went from 64kB lost to 6kB lost and the leak stacks were all from the curl_global_init.

Can you give this a try? I'll update the test suite and the docs.

[lathakris]

I was reading that curl_global_cleanup also has to be done ? should this be done after curl_easy_init() ?

[jasoncwik]

curl_global_cleanup should be done once at the end of your program.

[lathakris]

Sorry what I meant is should the curl_global_cleanup be done after calling curl_easy_cleanup. I will try this your suggestion.

[chandran-uday]

Is there any test samples available to call curl_ossl_init()?