`test_sign_from_pem_path_with_passphrase` fails intermittently with error `cryptography.exceptions.InvalidSignature`

[anjalirai-intel]

Description of the problem

We had seen intermittent failure for test_sign_from_pem_path_with_passphrase on AlmaLinux 9 and CentOS Stream 8, we tried to reproduce it by running the test for 50 times on both distro's but we were not able to reproduce it.

This test has again failed intermittently on Ubuntu 22.04 distro.

7-8 nightly have run on the original chroot PR and this is the first time we are seeing this issue on Ubuntu22.04 and in the past also we had seen this issue only once on Almalinux9 and CentOS Stream 8

Steps to reproduce

Run python3 -m pytest -v --junit-xml tests.xml tests/test_sgx_sign.py

Expected results

The test should pass successfully

Actual results

We were not able to reproduce so, attaching jenkins log

Ubuntu 22.04:

___________________ test_sign_from_pem_path_with_passphrase ____________________
tests/test_sgx_sign.py:80: in test_sign_from_pem_path_with_passphrase
    verify_signature(data, exponent, modulus, signature, key_file, passphrase)
tests/test_sgx_sign.py:50: in verify_signature
    public_key.verify(signature_bytes, data, padding.PKCS1v15(), hashes.SHA256())
/usr/lib/python3/dist-packages/cryptography/hazmat/backends/openssl/rsa.py:562: in verify
    return _rsa_sig_verify(
/usr/lib/python3/dist-packages/cryptography/hazmat/backends/openssl/rsa.py:283: in _rsa_sig_verify
    raise InvalidSignature
E   cryptography.exceptions.InvalidSignature

CentOS Stream 8:

___________________ test_sign_from_pem_path_with_passphrase ____________________
tests/test_sgx_sign.py:80: in test_sign_from_pem_path_with_passphrase
    verify_signature(data, exponent, modulus, signature, key_file, passphrase)
tests/test_sgx_sign.py:50: in verify_signature
    public_key.verify(signature_bytes, data, padding.PKCS1v15(), hashes.SHA256())
/usr/lib64/python3.6/site-packages/cryptography/hazmat/backends/openssl/rsa.py:464: in verify
    self._backend, padding, algorithm, self, signature, data
/usr/lib64/python3.6/site-packages/cryptography/hazmat/backends/openssl/rsa.py:250: in _rsa_sig_verify
    raise InvalidSignature
E   cryptography.exceptions.InvalidSignature

Gramine commit hash

a3694193649f8dccc9cb6eb349d092e65b686c5c

[dimakuv]

This feels like a concurrency issue. We have two tests for SGX signing keys:

• https://github.com/gramineproject/gramine/blob/7e260b175b7afa48af1dc55b754f52361e6c971f/tests/test_sgx_sign.py#L56
• https://github.com/gramineproject/gramine/blob/7e260b175b7afa48af1dc55b754f52361e6c971f/tests/test_sgx_sign.py#L70

We run them using classic PyTest:

• https://github.com/gramineproject/gramine/blob/7e260b175b7afa48af1dc55b754f52361e6c971f/.ci/lib/stage-test.jenkinsfile#L58-L68

And these two tests use the same file name:

• https://github.com/gramineproject/gramine/blob/7e260b175b7afa48af1dc55b754f52361e6c971f/tests/test_sgx_sign.py#L20

So I think the tests run in parallel and sometimes collide (the first test creates a key.pem without a passphrase, and the second test verifies this file, expecting a passphrase, and fails).

Though I don't understand why PyTest runs them in parallel. If I read their documentation correctly, PyTest doesn't parallelize tests by default? Anyway, need consultation from @woju and/or @oshogbo.

[anjalirai-intel]

Today, for the first time, test_sign_from_pem_path is failed on Debian 12

Error:

___________________________ test_sign_from_pem_path ____________________________
tests/test_sgx_sign.py:64: in test_sign_from_pem_path
    verify_signature(data, exponent, modulus, signature, key_file)
tests/test_sgx_sign.py:50: in verify_signature
    public_key.verify(signature_bytes, data, padding.PKCS1v15(), hashes.SHA256())
/usr/lib/python3/dist-packages/cryptography/hazmat/backends/openssl/rsa.py:569: in verify
    _rsa_sig_verify(
/usr/lib/python3/dist-packages/cryptography/hazmat/backends/openssl/rsa.py:325: in _rsa_sig_verify
    raise InvalidSignature
E   cryptography.exceptions.InvalidSignature