Python/SDK: Critical errors with pyo3

[gaby]

I have notice that when a server is running for +1 day using the infisical sdk errors will start showing up from pyo3 complaining about "No file handlers available". The only way to fix this is to restart the container running the python server.

This issue didn't happen with https://github.com/Infisical/infisical-python

Example Code (Throws error):

import time
from infisical_client import InfisicalClient, EncryptSymmetricOptions, DecryptSymmetricOptions

while True:
    #client = InfisicalClient()
    key = InfisicalClient().createSymmetricKey()
    encryptOptions = EncryptSymmetricOptions(
        key=key,
        plaintext="Infisical is awesome!"
    )

    encryptedData = InfisicalClient().encryptSymmetric(encryptOptions)

    decryptOptions = DecryptSymmetricOptions(
        ciphertext=encryptedData.ciphertext,
        iv=encryptedData.iv,
        tag=encryptedData.tag,
        key=key
    )

    decryptedString = InfisicalClient().decryptSymmetric(decryptOptions)
    print(f"Key={key}, cipher={decryptedString}")

Example Code (Works as intended):

import time
from infisical_client import InfisicalClient, EncryptSymmetricOptions, DecryptSymmetricOptions

while True:
    client = InfisicalClient()
    key = client.createSymmetricKey()
    encryptOptions = EncryptSymmetricOptions(
        key=key,
        plaintext="Infisical is awesome!"
    )

    encryptedData = client.encryptSymmetric(encryptOptions)

    decryptOptions = DecryptSymmetricOptions(
        ciphertext=encryptedData.ciphertext,
        iv=encryptedData.iv,
        tag=encryptedData.tag,
        key=key
    )

    decryptedString = client.decryptSymmetric(decryptOptions)
    print(f"Key={key}, cipher={decryptedString}")

@DanielHougaard Any ideas?

[varonix0]

Hi @gaby, sorry for the delayed response. Are you able to share a stack trace or a more complete error? Is the python server running on Windows by any chance? I remember reading about an issue like this specifically on Windows machines/servers.

If you're running on Linux, can you please share more information about the distribution / image you're using?

[gaby]

@DanielHougaard I have updated the example code, it only happens if I create an InfisicalClient per call. If I do a global one this doesn't happen.

Stack Trace:

thread '<unnamed>' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:447:13:
OS can't spawn worker thread: Resource temporarily unavailable (os error 11)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "/home/ubuntu/Desktop/git/fix/infisical-test.py", line 22, in <module>
    decryptedString = InfisicalClient().decryptSymmetric(decryptOptions)
  File "/usr/local/lib/python3.10/dist-packages/infisical_client/infisical_client.py", line 87, in decryptSymmetric
    result = self._run_command(Command(decrypt_symmetric=options))
  File "/usr/local/lib/python3.10/dist-packages/infisical_client/infisical_client.py", line 32, in _run_command
    response_json = self.inner.run_command(json.dumps(command.to_dict()))
pyo3_runtime.PanicException: OS can't spawn worker thread: Resource temporarily unavailable (os error 11)

[varonix0]

That's interesting, thank you for the stack traces. I'm leaning towards it being a cleanup issue. The Python garbage collector should be freeing up the resources being used by the client at the end of the loop, but it appears that something is going wrong there. I'll need to look deeper into this before I can give you a detailed rundown of what's going wrong. I'll keep you in the loop!

Thank you for reporting this, @gaby

[gaby]

@DanielHougaard Any update on this? I notice it still happens randomly using the "Working Example" above in tbe latest SDK version.

I'm not sure if its a bug in pyo3_runtime or the way the SDK is using it.

[gaby]

@DanielHougaard Asking ChatGPT I found two issues. First one is there's no limits being set for tokio threads:

use tokio::runtime::Builder;

#[tokio::main(flavor = "multi_thread", worker_threads = 4)] // Adjust the number of worker threads as needed
async fn main() {
    // Your code here
}

Second issue is the Rust code is using async with pyo3 and it should be using pyo3-asyncio https://github.com/awestlake87/pyo3-asyncio

Example

#[pymethods]
impl InfisicalClient {
    #[pyo3(text_signature = "($self, command_input)")]
    fn run_command<'a>(&'a mut self, command_input: String) -> PyResult<&'a PyAny> {
        Python::with_gil(|py| {
            future_into_py(py, async move {
                match run_command(&mut self.client, &command_input).await {
                    Ok(result) => Ok(result),
                    Err(e) => Err(pyo3::exceptions::PyRuntimeError::new_err(format!("{:?}", e))),
                }
            })
        })
    }
}

async fn run_command(client: &mut JsonClient, input_str: &str) -> Result<String, Box<dyn std::error::Error>> {
    client.run_command(input_str).await
}

[gaby]

I have moved on to use something else. This library can't handle high concurrency or be used from an HTTP handler.

[gaby]

Just an FYI, because of this issue we had to completely move off Infisical.

[varonix0]

HI @gaby, apologies for the delayed response here.

We have recently rolled out a native Python SDK (https://github.com/Infisical/python-sdk-official), which should resolve this issue. The new SDK is not dependent on Rust at all, and will run just like any Python library.

[gaby]

@DanielHougaard Thanks, any plans to re-add the encryption utils to it ?

[varonix0]

Hey @gaby, you're right that those are currently missing from the new SDK. Let me bring it by the team!

[varonix0]

Hi @gaby, after talking with the team about it, we've decided to not add the cryptography methods to the new native SDK. The reason for this, is because it's not proprietary or strictly related to Infisical, and the same can be achieved by using an external package which handles encryption.

Instead, we're going to be adding the Infisical KMS into our SDK! However a deadline for this hasn't been set yet, but it's on our roadmap for the Python SDK.

[gaby]

@DanielHougaard Sounds good, I will keep track of the releases.