TLSF memory allocator. Less free flash, moar free ram.

[DrZlo13]

What's new

• Heap4 allocator is replaced by tlsf.
• Moar free heap, less free flash.

Verification

• Run unit_tests
• In general, everything is affected.

Checklist (For Reviewer)

• [ ] PR has description of feature/bug or link to Confluence/Jira task
• [ ] Description contains actions to verify feature/bugfix
• [ ] I've built this code, uploaded it to the device and verified feature/bugfix

[github-actions[bot]]

PVS-Studio report for commit 3a4bff82:

• Report

[github-actions[bot]]

Compiled f7 firmware for commit cab517fb:

• 📦 Update package
• 📥 DFU file
• ☁️ Web/App updater
• 📊 Size report

[CookiePLMonster]

Could maybe aligned_malloc be removed in favour of aligned_alloc while you're breaking API anyway? Less symbols and coincidentally also better conformance with the C standard, especially now that non-standard aligned_free is gone: https://en.cppreference.com/w/c/memory/aligned_alloc

Or does definining standard C symbols come with other gotchas?

EDIT: The order of parameters is swapped between the two, that's nasty 😑

[Willy-JL]

perhaps as suggested in #3574 and on discord contributor chat, could have along malloc() also a malloc_nothrow() that does not crash on size=0 or out of memory? i get wanting to keep strict standards to make sure things dont go wrong due to the nature of embedded systems, but also maybe it should be availableas an option. not the default, but available if necessary

[CookiePLMonster]

perhaps as suggested in #3574 and on discord contributor chat, could have along malloc() also a malloc_nothrow() that does not crash on size=0 or out of memory? i get wanting to keep strict standards to make sure things dont go wrong due to the nature of embedded systems, but also maybe it should be availableas an option. not the default, but available if necessary

Ever since we discussed it, I had a realization. Realistically, this would only work for very temporary allocations, maybe even with the scheduler disabled, for a simple reason - if the OS is so close to OOM so a nothrow allocator fails (gracefully), one of the two can happen:

1. nothrow allocator succeeds, but now we're near OOM, and something else in the firmware is likely going to try allocate memory soon-ish and crash.
2. nothrow allocator doesn't succeed because we're past OOM. The same scenario happens - something else in the firmware is likely going to crash soon, and the nothrow allocator has only bought us some time.

Or am I missing some scenario, or overestimating how many "random" allocations the firmware is doing when doing its thing?

[Willy-JL]

there could 100% be some better way to handle this and is probably a PEBKAC situation, but the first thing that came to mind is reading huge files trying to get the whole content into ram. user selects a file, get filesize, try to malloc, it crashes. with nothrow, can say "file too big" or something like that. the simplest alternative i see is checking the filesize against free ram (or rather max heap block), but that could get clunky

[CookiePLMonster]

Ah yes, for graceful error handling situations this could be OK. I was thinking about a situation of "if this is too much, try to allocate a little bit less and try again" - which would then be asking for OOM on something else.

[DrZlo13]

There was a specific use case that is now covered by memmgr_heap_walk_blocks.

[Willy-JL]

yep, walking the blocks solves the part that noproto was interested in for mfkey if i remember correctly. still, a malloc_nothrow() could be a nice addition. though it is nothing that couldnt be solved with a max heap block size check, so i understand if it's a no in the end

[CookiePLMonster]

Max heap size check is prone to a race though, unless the program checks the heap size and mallocs with the memory manager locked. A nothrow alloc abstracts this detail away from the program.

~~That said, if the program needs to be memory-layout-aware enough to check for the max heap size, it can also just lock the heap for this operation, especially since the heap lock is recursive, so allocating with a held lock is OK.~~

EDIT: Nevermind, heap lock/unlock are not public symbols.

[DrZlo13]

@Willy-JL @CookiePLMonster First of all, I personally don't see how malloc_nothrow can be used at all. Iterate with decreasing size until it succeeds? Second, I don't understand how it can be used safely. You need to leave something around 20k for the OS. Third, we prefer not to give a loaded gun to anyone who is not smart enough to make and load it themselves.

(Hint: you can use block_walk and malloc while the kernel is stopped and even inside the critical section. Do not forget to start the kernel back, or exit from critical section to properly shoot your own leg off).

[CookiePLMonster]

(Hint: you can use block_walk and malloc while the kernel is stopped and even inside the critical section. Do not forget to start the kernel back, or exit from critical section to properly shoot your own leg off).

Yeah, that's what I meant by locking the heap, as memmgr_lock and _unlock are effectively that. Suspending/resuming tasks is re-entrant so it's safe to suspend it twice (once from the user code, then again from malloc).

[Sameesunkaria]

💛