How to handle too big of a memory allocation?

[assafmo]

We have HeapMaxSize set to 10 MiB. When we try to allocate more than 10 MiB we get SIGILL. Is there a nice way to catch and recover from these type of rogue allocations?

cc @reuvenpo @Cashmaney

[volcano0dr]

You can try two methods: 1.sgx_std::alloc::set_alloc_error_hook. Custom memory allocation error handling function.

2.sgx_signal::exception::register_exception Register exception handling function to capture SIGILL signal. example: https://github.com/apache/incubator-teaclave-sgx-sdk/blob/master/samplecode/unit-test/enclave/src/test_exception.rs

Although you can capture the event of memory allocation failure, but because there is not enough memory, the handler may raise exception again.

[reuvenpo]

That's one of our concerns... trying to recover from out-of-memory can double-fault, and in both cases, where does the code return to after the signal handler returns?

[volcano0dr]

The registered signal handler can return two states:

1.ContinueType:Search Continue searching for the next registered signal handler.

2.ContinueType::Execution Continue to execute the instruction that caused the exception.

If the exception is not handled on the handler chain (all return ContinueType:Search), the default handler will inject panic, and we can catch unwind.

config.xml:

<HeapMaxSize>0x80000</HeapMaxSize>

sample code:

use std::alloc;
use std::backtrace::{self, PrintFormat};
use std::panic;
use std::vec::Vec;
use sgx_signal::ContinueType;
use sgx_signal::exception;
use sgx_types::sgx_exception_info_t;

#[no_mangle]
pub extern "C" fn ecall_entry() {
    let _ = backtrace::enable_backtrace("enclave.signal.so", PrintFormat::Full);
    let handler = {
        move |_info: &mut sgx_exception_info_t| {
            ContinueType::Search
        }
    };
    
    alloc::set_alloc_error_hook(|layout| {
        println!("memory allocation of {} bytes failed", layout.size());
        unsafe { std::intrinsics::abort() };
    });

    let _h = exception::register_exception(true, handler);
    
    panic::catch_unwind(|| {
        test_alloc_memory()
    }).ok();

    let mut success =  vec![0_u8; 0x20000];
    success[0] = 1;
}

#[no_mangle]
#[inline(never)]
fn test_alloc_memory() {
    let _failed = vec![0_u8; 0x100000];
}

@reuvenpo hope it can help you.

[reuvenpo]

Oh wow, thanks @volcano0dr this is extremely helpful! We'll adapt this to our needs in our codebase in the next few days, i'll let you know whether it works well for us :)

[assafmo]

@volcano0dr So the exception::register_exception(true, handler) does not invoke handler on memory allocation of 2228224 bytes failed SIGILL: illegal instruction, what should I do?

[assafmo]

Is there a way to recover with sgx_std::alloc::set_alloc_error_hook?

[assafmo]

I threw panic inside sgx_std::alloc::set_alloc_error_hook and it got to the catch_unwind clause :sweat_smile:

[volcano0dr]

@assafmo Before registering the exception handler, has the alloc::set_alloc_error_hook function been called to set the allocation failure handler?

alloc::set_alloc_error_hook(|layout| {
        println!("memory allocation of {} bytes failed", layout.size());
        unsafe { std::intrinsics::abort() };
    });

The default handler for memory allocation failure will set the enclave status to ENCLAVE_CRASHED, and the enclave will no longer catch exception signals in this state.

[reuvenpo]

Yeah so we found out that the enclave will behave nicely and remain useful if you set sgx_std::alloc::set_alloc_error_hook to cause a panic. This triggers a nice and orderly unwind, which we then catch and gracefully handle :)

[assafmo]

@volcano0dr @dingelish Does this also work if the enclave passes the stack size limit? Or should we also implement protections against stack over-allocations somehow? If yes, how should we approach this?

[volcano0dr]

@assafmo If the enclave passes the stack size limit, EALL will return SGX_ERROR_STACK_OVERRUN.

Enclave thread context memory layout: guard page | stack | guard page | TCS | SSA | guard page | TLS/TD

When the stack is overallocated, the enclave will receive an exception, and the exception handler will check whether the rsp at the time of the exception is in the stack range. If the rsp is not in the stack boundary, set the enclave status to ENCLAVE_CRASHED, and ECALL will return with SGX_ERROR_STACK_OVERRUN.

[assafmo]

Thanks for the prompt response @volcano0dr! Is there any way to recover from this, like we did with sgx_std::alloc::set_alloc_error_hook for heap over-allocation?

[volcano0dr]

I haven’t found a simple way to recover from this, because the stack has been overused, and exceptions may occur when running any code. One possible way is to use another additional stack, just as the exception handler uses a separate stack.