Memory usage increase in multi-threaded env

[tshmul]

I saw a permanent memory usage increase while keep performing AES/CBC encryption in multi-threaded execution. It seems the memory released when execution finished. Am I not using the encryption correctly? or is it known to not support multi-threaded, and I should add a lock for encryption/decryption? (I saw https://github.com/Legrandin/pycryptodome/issues/12 and it seems pycryptodome support mutli-thread)

Example code:

import os
import concurrent.futures
from Cryptodome.Cipher import AES


def encrypt_loop(text, key, iv):
    for i in range(50000):
        aes = AES.new(key, AES.MODE_CBC, iv=iv)
        ctext = aes.encrypt(text)


if __name__ == '__main__':
    NUMBER_OF_THREADS = 20
    text = os.urandom(16)
    key = os.urandom(32)
    iv = b'\0' * 16
    with concurrent.futures.ThreadPoolExecutor(NUMBER_OF_THREADS) as executor:
        for t in range(NUMBER_OF_THREADS):
            executor.submit(encrypt_loop, text, key, iv)

        executor.shutdown()

    import pdb;pdb.set_trace()

Environment details:

• python 3.8.10 (reproduced also with python 3.7.5)
• pycryptodome 3.10.1
• windows machine.

Thanks, Tomer

[tshmul]

After investigating this issue, I found out this issue occurs since python does not call del of SmartPointer immediately when exiting scope. This behavior is known, see for example https://stackoverflow.com/questions/1481488/what-is-the-del-method-and-how-do-i-call-it/2452895#2452895.

I would like to suggest using a context manager object (using with) for all pycryptodome objects that require releasing resources, we can make it optional to avoid breaking the API, and advise users to use this method.

@Legrandin WDYT? make sense? should I open a branch for this change?

[Legrandin]

Thanks for finding this issue. This is fixed with https://github.com/Legrandin/pycryptodome/commit/d2c925ce90fd462cfae3c3a1ad60b92800a2704e.

It took a while, but I discovered that, in reality, this issue is not caused by neither multithreading nor the use of __del__. Instead, objects such as CBC used references to internal bounded methods for state tracking, and that clearly threw off the garbage collector, which didn't finalize unused objects anymore.

[Legrandin]

Fixed in v3.17