Botan in JVM + SoftHSM

[antoinelochet]

Hello, We use SoftHSM for testing in our Java application. We would like to have Botan to test our application through SoftHSM becuase of the PQC support which do not exist (and will probably never exist) in OpenSSL 1.1 We encounter a SIGSEGV when SoftHSM is stopping with the JVM and is destructing the Botan instance:

Current thread (0x00007fe48811a570):  VMThread "VM Thread" [stack: 0x00007fe48c6b0000,0x00007fe48c7b0000] [id=5978]

Stack: [0x00007fe48c6b0000,0x00007fe48c7b0000],  sp=0x00007fe48c7ae448,  free space=1017k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.6+0x16173c]
C  [libbotan-2.so.19+0x7e0e21]  Botan::deallocate_memory(void*, unsigned long, unsigned long)+0x36
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x13b428]  Botan::secure_allocator<unsigned long>::deallocate(unsigned long*, unsigned long)+0x2c
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x13b386]  std::allocator_traits<Botan::secure_allocator<unsigned long> >::deallocate(Botan::secure_allocator<unsigned long>&, unsigned long*, unsigned long)+0x2b
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x13b186]  std::_Vector_base<unsigned long, Botan::secure_allocator<unsigned long> >::_M_deallocate(unsigned long*, unsigned long)+0x32
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x13b03a]  std::_Vector_base<unsigned long, Botan::secure_allocator<unsigned long> >::~_Vector_base()+0x3e
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x13b07f]  std::vector<unsigned long, Botan::secure_allocator<unsigned long> >::~vector()+0x41
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x13acfe]  Botan::BigInt::Data::~Data()+0x18
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x13ad26]  Botan::BigInt::~BigInt()+0x24
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x15c15e]  BotanSymmetricAlgorithm::~BotanSymmetricAlgorithm()+0x124
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x13569c]  BotanAES::~BotanAES()+0x2a
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x1356b8]  BotanAES::~BotanAES()+0x18
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x12f3d8]  CryptoFactory::recycleSymmetricAlgorithm(SymmetricAlgorithm*)+0x28
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x1666ef]  SecureDataManager::~SecureDataManager()+0x7d
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x1668fe]  SecureDataManager::~SecureDataManager()+0x18
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x188a0f]  Token::~Token()+0x47
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x188a46]  Token::~Token()+0x18
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x188494]  Slot::~Slot()+0x3a
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x1884b0]  Slot::~Slot()+0x18
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x18692b]  SlotManager::~SlotManager()+0x137
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x1869ec]  SlotManager::~SlotManager()+0x18
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0xf1f43]  SoftHSM::~SoftHSM()+0xb1
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0xf2022]  SoftHSM::~SoftHSM()+0x18
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x1250a2]  std::default_delete<SoftHSM>::operator()(SoftHSM*) const+0x28
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x126d56]  std::unique_ptr<SoftHSM, std::default_delete<SoftHSM> >::~unique_ptr()+0x52

I have tried :

• several compilation configurations
• removing the delete instructions in the SoftHSM code in the destructor of the SecureDataManager to see if it would do anything
• changing the Botan::BigInt in BotanSymmetricAlgorithm to Botan::BigInt* and even std::unique_ptr<Botan::BigInt> but, alas, the same error occurs. Sometimes elsewhere :

---------------  T H R E A D  ---------------

Current thread (0x00007f410411a570):  VMThread "VM Thread" [stack: 0x00007f40e8700000,0x00007f40e8800000] [id=4819]

Stack: [0x00007f40e8700000,0x00007f40e8800000],  sp=0x00007f40e87fe608,  free space=1017k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.6+0x16173c]
C  [libbotan-2.so.19+0x7e0e21]  Botan::deallocate_memory(void*, unsigned long, unsigned long)+0x36
C  [libbotan-2.so.19+0x40053e]  Botan::secure_allocator<unsigned char>::deallocate(unsigned char*, unsigned long)+0x2c
C  [libbotan-2.so.19+0x3ffb58]  std::allocator_traits<Botan::secure_allocator<unsigned char> >::deallocate(Botan::secure_allocator<unsigned char>&, unsigned char*, unsigned long)+0x2b
C  [libbotan-2.so.19+0x3ff504]  std::_Vector_base<unsigned char, Botan::secure_allocator<unsigned char> >::_M_deallocate(unsigned char*, unsigned long)+0x32
C  [libbotan-2.so.19+0x3feda8]  std::_Vector_base<unsigned char, Botan::secure_allocator<unsigned char> >::~_Vector_base()+0x3a
C  [libbotan-2.so.19+0x3feded]  std::vector<unsigned char, Botan::secure_allocator<unsigned char> >::~vector()+0x41
C  [libbotan-2.so.19+0x718132]  Botan::HMAC_DRBG::~HMAC_DRBG()+0x2e
C  [libbotan-2.so.19+0x71816a]  Botan::HMAC_DRBG::~HMAC_DRBG()+0x18
C  [libbotan-2.so.19+0x7426fa]  std::default_delete<Botan::Stateful_RNG>::operator()(Botan::Stateful_RNG*) const+0x28
C  [libbotan-2.so.19+0x7425b4]  std::unique_ptr<Botan::Stateful_RNG, std::default_delete<Botan::Stateful_RNG> >::~unique_ptr()+0x52
C  [libbotan-2.so.19+0x741c48]  Botan::AutoSeeded_RNG::~AutoSeeded_RNG()+0x2e
C  [libbotan-2.so.19+0x741c70]  Botan::AutoSeeded_RNG::~AutoSeeded_RNG()+0x18
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x155045]  BotanRNG::~BotanRNG()+0x83
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x1550d4]  BotanRNG::~BotanRNG()+0x18
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x1368e2]  BotanCryptoFactory::~BotanCryptoFactory()+0xba
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x136974]  BotanCryptoFactory::~BotanCryptoFactory()+0x18
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x126eae]  std::default_delete<BotanCryptoFactory>::operator()(BotanCryptoFactory*) const+0x28
C  [libsofthsm-develop-with-kdf-and-date-encryption.so+0x126cf2]  std::unique_ptr<BotanCryptoFactory, std::default_delete<BotanCryptoFactory> >::~unique_ptr()+0x52

Botan is compiled with --extra-cxxflags="-O0" --debug-mode --with-debug-info --without-stack-protector options passed to the configure.py script. Maybe some duplicate/useless options in here :)

Do you have any idea of what is causing this issue ?

OS: Debian 11.6 on WSL2 Java version:

openjdk version "17.0.6" 2023-01-17
OpenJDK Runtime Environment (build 17.0.6+10-Debian-1deb11u1)
OpenJDK 64-Bit Server VM (build 17.0.6+10-Debian-1deb11u1, mixed mode, sharing)

(The same behavior happens with Java 20 also) SoftHSM 2.6.1