apache
sched/tcb: use shared group for kthreads
Summary
Current task group data is embedded in the task TCB, this is also used for kernel threads, which seems can share the group as they all live in kernel space.
This patch uses tcb_s for kthreads with a shared group instance. Thus the overhead is reduced when more kthreads are in use.
Memory usage of group instance, i.e. sizeof(task_group_s), on rv-virt device:
| config | size |
|---|---|
| nsh | 720 |
| nsh64 | 1160 |
| knsh32 | 248 |
| knsh64 | 440 |
Kernel memory footprints collected from attached logs:
| config | kthreads # | upstream | patched | saved |
|---|---|---|---|---|
| nsh | 1 | 7044 | 7044 | 0 |
| nsh64 | 1 | 10344 | 10344 | 0 |
| knsh32 | 2 | 10348 | 9660 | 588 |
| knsh64 | 2 | 13304 | 12168 | 1136 |
The savings are beyond sizeof(task_group_s) due to subordinate data of groups.
Impact
This may affect all configs.
Testing
- QEMU 6.2 on Ubuntu 22.04 with configs
nsh,nsh64,knsh32andknsh64. See logs-12320.tar.gz for more details:- the
embeddedfolder has logs from upstream version. - the
kthreadsfolder has logs from patched version.
- the
- Github CI checks
@anchao, let's discuss here as these comments are visible in gh tool as well.
Can you teach your opinion about the MM_TLSF_MANAGER? It seems having O(1) alloc() and works in flat build mode. I added *.tlsf.log files for your review.
@anchao, let's discuss here as these comments are visible in
ghtool as well.Can you teach your opinion about the
MM_TLSF_MANAGER? It seems having O(1)alloc()and works in flat build mode. I added*.tlsf.logfiles for your review.
Although TLSF has more advantages than Best fit in terms of time complexity O(1), it will cause serious memory fragmentation issues in aging and persistence tests.
I have added an attachment containing the test results of different allocators (Sheet from @XuNeo Thanks), You can generate a chart and observe the memory trend (This chart may not show the worst case scenario of TLSF) 20221110_mem_test.xlsx
@anchao, thanks for sharing the details.
Here I am trying to keep task_tcb_s for userspace tasks but use tcb_s for kthreads as suggested by @xiaoxiang781216 and it looks working for nsh64, I still check other configs. Is this approach fine for you?
@anchao, thanks for sharing the details.
Here I am trying to keep
task_tcb_sfor userspace tasks but usetcb_sfor kthreads as suggested by @xiaoxiang781216 and it looks working fornsh64, I still check other configs. Is this approach fine for you?
Sounds good, could you update this PR, I'd like to review the code directly
Please fix this failure:
In function 'nxthread_create',
inlined from 'kthread_create_with_stack' at task/task_create.c:254:10:
Error: task/task_create.c:97:18: error: array subscript 'struct task_tcb_s[0]' is partly outside array bounds of 'unsigned char[368]' [-Werror=array-bounds]
97 | tcb->cmn.flags = ttype | TCB_FLAG_FREE_TCB;
| ^
In file included from task/task_create.c:33:
task/task_create.c:88:9: note: object of size 368 allocated by 'zalloc'
88 | tcb = kmm_zalloc(ret);
| ^~~~~~~~~~
Error: task/task_create.c:111:7: error: array subscript 'struct task_tcb_s[0]' is partly outside array bounds of 'unsigned char[368]' [-Werror=array-bounds]
111 | pid = tcb->cmn.pid;
| ~~~~^~~~~~~~~~~~~~
task/task_create.c:88:9: note: object of size 368 allocated by 'zalloc'
88 | tcb = kmm_zalloc(ret);
| ^~~~~~~~~~
In function 'nxthread_create',
inlined from 'kthread_create_with_stack' at task/task_create.c:254:10,
inlined from 'kthread_create' at task/task_create.c:285:10:
Error: task/task_create.c:97:18: error: array subscript 'struct task_tcb_s[0]' is partly outside array bounds of 'unsigned char[368]' [-Werror=array-bounds]
97 | tcb->cmn.flags = ttype | TCB_FLAG_FREE_TCB;
| ^
task/task_create.c:88:9: note: object of size 368 allocated by 'zalloc'
88 | tcb = kmm_zalloc(ret);
| ^~~~~~~~~~
Error: task/task_create.c:111:7: error: array subscript 'struct task_tcb_s[0]' is partly outside array bounds of 'unsigned char[368]' [-Werror=array-bounds]
111 | pid = tcb->cmn.pid;
| ~~~~^~~~~~~~~~~~~~
task/task_create.c:88:9: note: object of size 368 allocated by 'zalloc'
88 | tcb = kmm_zalloc(ret);
| ^~~~~~~~~~
cc1: all warnings being treated as errors
make[1]: *** [Makefile:61: task_create.o] Error 1
make[1]: Target 'libsched.a' not remade because of errors.
@acassis I am wondering that is this GCC 12 issue, just pushed a version using $progma to see if it can be fixed. Please teach if there are better fixes.
@yf13 adding #pragma in the generic code is not a good idea, maybe you can try to fix the structure to avoid the issue, please see here: https://github.com/micropython/micropython/issues/7064
He create a union to storage the right size
@yf13 adding #pragma in the generic code is not a good idea, maybe you can try to fix the structure to avoid the issue, please see here: micropython/micropython#7064
@acassis thanks, will remove #pragma later after passing all checks.
Now case test_ltp_interfaces_lio_listio_2_1 still fails, need more investigation about it, hints are highly welcomed.
@acassis, @xiaoxiang781216 and @anchao, the issues have been fixed, please review when free.
@xiaoxiang781216 I removed volatile uintptr_t usage in nx_start.c and the out-bounds warnings come back for sim/sotest as per the CI log from Linux(sim-02).
Please let me know if there are better ways or I shall add them back with comments like below:
{
- tcb = (FAR struct task_tcb_s *)&g_idletcb[i];
+ /* To shut up gcc warnings for `sim/sotest` */
+
+ volatile uintptr_t p = (uintptr_t)&g_idletcb[i];
+ tcb = (FAR struct task_tcb_s *)p;
@xiaoxiang781216 I removed
volatile uintptr_tusage innx_start.cand theout-boundswarnings come back forsim/sotestas per the CI log from Linux(sim-02).Please let me know if there are better ways or I shall add them back with comments like below:
{ - tcb = (FAR struct task_tcb_s *)&g_idletcb[i]; + /* To shut up gcc warnings for `sim/sotest` */ + + volatile uintptr_t p = (uintptr_t)&g_idletcb[i]; + tcb = (FAR struct task_tcb_s *)p;
why not change type of tcb to FAR struct tcb_s * and remove the cast?
@xiaoxiang781216 thanks, I did that way before but stopped to reduce number of changing lines. now let's go back to avoid warnings.
Is the CI system still broken? This Linux(sim-02) log complains below for sim/sqlite:
/github/workspace/sources/apps/database/sqlite/sqlite/configure: line 5204: /usr/bin/file: No such file or directory
/github/workspace/sources/apps/database/sqlite/sqlite/configure: line 10376: tclsh: command not found
@xiaoxiang781216 thanks, I did that way before but stopped to reduce number of changing lines. now let's go back to avoid warnings.
Is the CI system still broken? This Linux(sim-02) log complains below for
sim/sqlite:/github/workspace/sources/apps/database/sqlite/sqlite/configure: line 5204: /usr/bin/file: No such file or directory /github/workspace/sources/apps/database/sqlite/sqlite/configure: line 10376: tclsh: command not found
sqlite patch generate this ci failure, which is fixed by https://github.com/apache/nuttx/pull/12374
@xiangxiang781216 not sure if CI checks are back to normal? I saw issues with some LTP tests thus checked them with local build:
nsh> time ltp_behavior_WIFEXITED_1_1
Test PASSED
5.0100 sec
nsh> time ltp_behavior_WIFEXITED_1_2
Test PASSED
5.0100 sec
nsh> time ltp_interfaces_pthread_once_2_1
1.0100 sec
I've temporarily disabled them to see more issues.
@xiaoxiang781216 now the remaining issues are in this log:
Configuration/Tool: rv-virt/citest64
usrsocktest_basic_connect.c: In function 'teardown':
Error: usrsocktest_basic_connect.c:112:7: error: variable 'ret' set but not used [-Werror=unused-but-set-variable]
and
Configuration/Tool: esp32c3-generic/twai esp32c6-devkitm/twai esp32c6-devkitc/twai esp32h2-devkit/twai
Error: common/espressif/esp_twai.c:242:7: error: variable 'ret' set but not used [-Werror=unused-but-set-variable]
patch #12387 has been created for the latter.
@xiaoxiang781216 now the remaining issues are in this log:
Configuration/Tool: rv-virt/citest64 usrsocktest_basic_connect.c: In function 'teardown': Error: usrsocktest_basic_connect.c:112:7: error: variable 'ret' set but not used [-Werror=unused-but-set-variable]and
Configuration/Tool: esp32c3-generic/twai esp32c6-devkitm/twai esp32c6-devkitc/twai esp32h2-devkit/twai Error: common/espressif/esp_twai.c:242:7: error: variable 'ret' set but not used [-Werror=unused-but-set-variable]patch #12387 has been created for the latter.
@yf13 thanks, see the comment in the pr.
@xiaoxiang781216 now the remaining issues are in this log:
Configuration/Tool: rv-virt/citest64 usrsocktest_basic_connect.c: In function 'teardown': Error: usrsocktest_basic_connect.c:112:7: error: variable 'ret' set but not used [-Werror=unused-but-set-variable]
Fix here: https://github.com/apache/nuttx-apps/pull/2403
@xiaoxiang781216 it looks that the LTP timeouts in sim/citest still happens, so I disabled them again to see if there are remaining issues. Then a few CI attempts failed due to connectivity issues encountered during CI. Then the usrsocktest issues come back again in my last run.
So looks like that I have to wait until CI is normal to trigger another check.
@xiaoxiang781216 it looks that the LTP timeouts in
sim/citeststill happens, so I disabled them again to see if there are remaining issues. Then a few CI attempts failed due to connectivity issues encountered during CI. Then the usrsocktest issues come back again in my last run.So looks like that I have to wait until CI is normal to trigger another check.
@Donny9 is looking this problem.
@anchao, @xiaoxiang781216 looks like the CI system is okay now. Please give this patch a review when free and let me know if there is anything I can do.
@xiaoxiang781216 looks like our CI got connectivity issue:
Configuration/Tool: ucans32k146/se05x,CONFIG_ARM_TOOLCHAIN_GNU_EABI
ccuurrll:: ((2288)) FFaaiilleedd ttoo cocnonnencet ctto tgoi tghiutbh.ucbo.mc opmo rpto r4t4 3 4a4f3t earf t1e3r5 110365 2m4s4: mCso:n nCeocntinoenc ttiimoend toiumte
I forgot somewhere I saw other repositories using sub-module instead of download, not sure if it can reduce this kind of connectivity issues?
Looks like the Linux (sim-xxx) checks passed so our compilation warning fixes shall work now. I will trigger another check soon.
@xiaoxiang781216 latest push also address the issue below, but somehow I couldn't reply directly from my browser.
So basically latest update should have addressed all in-place comments, please help to review when free.
@yf13 Sorry I think this commit is having problems on Ox64, it says "hello not found": https://gist.github.com/lupyuen/ceac79d1c3807ef1c0d7942012e8e95d
## NuttX Mainline
## https://github.com/apache/nuttx/tree/9790248f9a78c518d53fe3e60953d652f9eebbce
+ /Users/luppy/riscv/ox64-tinyemu/temu root-riscv64.cfg
TinyEMU Emulator for Ox64 BL808 RISC-V SBC
NuttShell (NSH) NuttX-12.4.0
nsh> hello
nsh: hello: command not found
nsh> uname -a
NuttX 12.4.0 9790248f9a Jun 24 2024 08:50:35 risc-v ox64
nsh> ls -l /system/bin/hello
-r-xr-xr-x 603464 /system/bin/hello
It works OK when I revert the above commit: https://gist.github.com/lupyuen/aad99dabdbf82260833cfff01969cdb6
## Revert "sched/tcb: use shared group for kthreads"
## https://github.com/apache/nuttx/tree/b8e453e4c1a7df55caaa45ff7e5ad5d0690c440b
+ /Users/luppy/riscv/ox64-tinyemu/temu root-riscv64.cfg
TinyEMU Emulator for Ox64 BL808 RISC-V SBC
NuttShell (NSH) NuttX-12.4.0
nsh> uname -a
NuttX 12.4.0 b8e453e4c1 Jun 24 2024 09:06:18 risc-v ox64
nsh> hello
Hello, World!!
free is also crashing on Ox64: https://github.com/lupyuen/nuttx-ox64/actions/runs/9638022080/job/26578084327
nsh> free
raise_exception2: cause=13, tval=0x50409ab000000010, pc=0x50218bf4
riscv_exception: EXCEPTION: Load page fault. MCAUSE: 000000000000000d, EPC: 0000000050218bf4, MTVAL: 50409ab000000010
riscv_exception: Segmentation fault in PID 3: /system/bin/init
_assert: Assertion failed !nxmutex_is_hold(mutex): at file: misc/lib_mutex.c:199 task: /system/bin/init process: Kernel 0x8000004a
## 0x50218bf4 points to mm/mm_heap/mm_mallinfo.c:160
## Is the Heap corrupted? MTVAL seems to be shifted incorrectly by 32 bits
I'll turn on the Binary Loader logging to get more details. Any idea what might have caused this? Thanks!
UPDATE: QEMU knsh64 has the same problem when CONFIG_BOARD_LATE_INITIALIZE=y. Wonder why Late Init is causing this problem on Ox64 and QEMU? Hmmm... https://gist.github.com/lupyuen/9eb6526672fdeaadf04a57a87b7c1db9
## NuttX Mainline: https://github.com/apache/nuttx/tree/9790248f9a78c518d53fe3e60953d652f9eebbce
+ tools/configure.sh rv-virt:knsh64
+ qemu-system-riscv64 -semihosting -M virt,aclint=on -cpu rv64 -kernel nuttx -nographic
NuttShell (NSH) NuttX-12.4.0
nsh> hello
nsh: hello: command not found
nsh> uname -a
NuttX 12.4.0 9790248f9a-dirty Jun 24 2024 13:52:12 risc-v rv-virt
nsh> ls -l /system/bin/hello
-rwxrwxrwx 109336 /system/bin/hello
nsh> free
[ 12.699000] _assert: Current Version: NuttX 12.4.0 9790248f9a-dirty Jun 24 2024 13:52:02 risc-v
[ 12.699000] _assert: Assertion failed nodesize >= (1 << (((sizeof(struct mm_freenode_s)) > 0 && ((sizeof(struct mm_freenode_s)) & (sizeof(struct mm_freenode_s) - 1)) == 0) ? ((sizeof(struct mm_freenode_s)) & 0xffffffff00000000ul ? 32 + (((size_t)(sizeof(struct mm_freenode_s)) >> 32) & 0xffff0000 ? 16 + ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) & 0xff00 ? 8 + (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) & 0xf0 ? 4 + ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 4) & 0xc ? 2 + (((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 4) >> 2) & 0x2 ? 1 + ((((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 4) >> 2) >> 1) & 0x1 ? 1 : 0) : (((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 4) >> 2) & 0x1 ? 1 : 0)) : ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 4) & 0x2 ? 1 + (((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 4) >> 1) & 0x1 ? 1 : 0) : ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 4) & 0x1 ? 1 : 0))) : (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) & 0xc ? 2 + ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 2) & 0x2 ? 1 + (((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 2) >> 1) & 0x1 ? 1 : 0) : ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 2) & 0x1 ? 1 : 0)) : (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) & 0x2 ? 1 + ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 1) & 0x1 ? 1 : 0) : (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) & 0x1 ? 1 : 0)))) : ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) & 0xf0 ? 4 + (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 4) & 0xc ? 2 + ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 4) >> 2) & 0x2 ? 1 + (((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 4) >> 2) >> 1) & 0x1 ? 1 : 0) : ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 4) >> 2) & 0x1 ? 1 : 0)) : (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 4) & 0x2 ? 1 + ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 4) >> 1) & 0x1 ? 1 : 0) : (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 4) & 0x1 ? 1 : 0))) : ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) & 0xc ? 2 + (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 2) & 0x2 ? 1 + ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 2) >> 1) & 0x1 ? 1 : 0) : (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 2) & 0x1 ? 1 : 0)) : ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) & 0x2 ? 1 + (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 1) & 0x1 ? 1 : 0) : ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) & 0x1 ? 1 : 0))))) : (((size_t)(sizeof(struct mm_freenode_s)) >> 32) & 0xff00 ? 8 + ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) & 0xf0 ? 4 + (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 4) & 0xc ? 2 + ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 4) >> 2) & 0x2 ? 1 + (((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 4) >> 2) >> 1) & 0x1 ? 1 : 0) : ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 4) >> 2) & 0x1 ? 1 : 0)) : (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 4) & 0x2 ? 1 + ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 4) >> 1) & 0x1 ? 1 : 0) : (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 4) & 0x1 ? 1 : 0))) : ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) & 0xc ? 2 + (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 2) & 0x2 ? 1 + ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 2) >> 1) & 0x1 ? 1 : 0) : (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 2) & 0x1 ? 1 : 0)) : ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) & 0x2 ? 1 + (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 1) & 0x1 ? 1 : 0) : ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) & 0x1 ? 1 : 0)))) : (((size_t)(sizeof(struct mm_freenode_s)) >> 32) & 0xf0 ? 4 + ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 4) & 0xc ? 2 + (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 4) >> 2) & 0x2 ? 1 + ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 4) >> 2) >> 1) & 0x1 ? 1 : 0) : (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 4) >> 2) & 0x1 ? 1 : 0)) : ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 4) & 0x2 ? 1 + (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 4) >> 1) & 0x1 ? 1 : 0) : ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 4) & 0x1 ? 1 : 0))) : (((size_t)(sizeof(struct mm_freenode_s)) >> 32) & 0xc ? 2 + ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 2) & 0x2 ? 1 + (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 2) >> 1) & 0x1 ? 1 : 0) : ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 2) & 0x1 ? 1 : 0)) : (((size_t)(sizeof(struct mm_freenode_s)) >> 32) & 0x2 ? 1 + ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 1) & 0x1 ? 1 : 0) : (((size_t)(sizeof(struct mm_freenode_s)) >> 32) & 0x1 ? 1 : 0)))))) : ((sizeof(struct mm_freenode_s)) & 0xffff0000 ? 16 + (((sizeof(struct mm_freenode_s)) >> 16) & 0xff00 ? 8 + ((((sizeof(struct mm_freenode_s)) >> 16) >> 8) & 0xf0 ? 4 + (((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 4) & 0xc ? 2 + ((((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 4) >> 2) & 0x2 ? 1 + (((((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 4) >> 2) >> 1) & 0x1 ? 1 : 0) : ((((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 4) >> 2) & 0x1 ? 1 : 0)) : (((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 4) & 0x2 ? 1 + ((((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 4) >> 1) & 0x1 ? 1 : 0) : (((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 4) & 0x1 ? 1 : 0))) : ((((sizeof(struct mm_freenode_s)) >> 16) >> 8) & 0xc ? 2 + (((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 2) & 0x2 ? 1 + ((((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 2) >> 1) & 0x1 ? 1 : 0) : (((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 2) & 0x1 ? 1 : 0)) : ((((sizeof(struct mm_freenode_s)) >> 16) >> 8) & 0x2 ? 1 + (((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 1) & 0x1 ? 1 : 0) : ((((sizeof(struct mm_freenode_s)) >> 16) >> 8) & 0x1 ? 1 : 0)))) : (((sizeof(struct mm_freenode_s)) >> 16) & 0xf0 ? 4 + ((((sizeof(struct mm_freenode_s)) >> 16) >> 4) & 0xc ? 2 + (((((sizeof(struct mm_freenode_s)) >> 16) >> 4) >> 2) & 0x2 ? 1 + ((((((sizeof(struct mm_freenode_s)) >> 16) >> 4) >> 2) >> 1) & 0x1 ? 1 : 0) : (((((sizeof(struct mm_freenode_s)) >> 16) >> 4) >> 2) & 0x1 ? 1 : 0)) : ((((sizeof(struct mm_freenode_s)) >> 16) >> 4) & 0x2 ? 1 + (((((sizeof(struct mm_freenode_s)) >> 16) >> 4) >> 1) & 0x1 ? 1 : 0) : ((((sizeof(struct mm_freenode_s)) >> 16) >> 4) & 0x1 ? 1 : 0))) : (((sizeof(struct mm_freenode_s)) >> 16) & 0xc ? 2 + ((((sizeof(struct mm_freenode_s)) >> 16) >> 2) & 0x2 ? 1 + (((((sizeof(struct mm_freenode_s)) >> 16) >> 2) >> 1) & 0x1 ? 1 : 0) : ((((sizeof(struct mm_freenode_s)) >> 16) >> 2) & 0x1 ? 1 : 0)) : (((sizeof(struct mm_freenode_s)) >> 16) & 0x2 ? 1 + ((((sizeof(struct mm_freenode_s)) >> 16) >> 1) & 0x1 ? 1 : 0) : (((sizeof(struct mm_freenode_s)) >> 16) & 0x1 ? 1 : 0))))) : ((sizeof(struct mm_freenode_s)) & 0xff00 ? 8 + (((sizeof(struct mm_freenode_s)) >> 8) & 0xf0 ? 4 + ((((sizeof(struct mm_freenode_s)) >> 8) >> 4) & 0xc ? 2 + (((((sizeof(struct mm_freenode_s)) >> 8) >> 4) >> 2) & 0x2 ? 1 + ((((((sizeof(struct mm_freenode_s)) >> 8) >> 4) >> 2) >> 1) & 0x1 ? 1 : 0) : (((((sizeof(struct mm_freenode_s)) >> 8) >> 4) >> 2) & 0x1 ? 1 : 0)) : ((((sizeof(struct mm_freenode_s)) >> 8) >> 4) & 0x2 ? 1 + (((((sizeof(struct mm_freenode_s)) >> 8) >> 4) >> 1) & 0x1 ? 1 : 0) : ((((sizeof(struct mm_freenode_s)) >> 8) >> 4) & 0x1 ? 1 : 0))) : (((sizeof(struct mm_freenode_s)) >> 8) & 0xc ? 2 + ((((sizeof(struct mm_freenode_s)) >> 8) >> 2) & 0x2 ? 1 + (((((sizeof(struct mm_freenode_s)) >> 8) >> 2) >> 1) & 0x1 ? 1 : 0) : ((((sizeof(struct mm_freenode_s)) >> 8) >> 2) & 0x1 ? 1 : 0)) : (((sizeof(struct mm_freenode_s)) >> 8) & 0x2 ? 1 + ((((sizeof(struct mm_freenode_s)) >> 8) >> 1) & 0x1 ? 1 : 0) : (((sizeof(struct mm_freenode_s)) >> 8) & 0x1 ? 1 : 0)))) : ((sizeof(struct mm_freenode_s)) & 0xf0 ? 4 + (((sizeof(struct mm_freenode_s)) >> 4) & 0xc ? 2 + ((((sizeof(struct mm_freenode_s)) >> 4) >> 2) & 0x2 ? 1 + (((((sizeof(struct mm_freenode_s)) >> 4) >> 2) >> 1) & 0x1 ? 1 : 0) : ((((sizeof(struct mm_freenode_s)) >> 4) >> 2) & 0x1 ? 1 : 0)) : (((sizeof(struct mm_freenode_s)) >> 4) & 0x2 ? 1 + ((((sizeof(struct mm_freenode_s)) >> 4) >> 1) & 0x1 ? 1 : 0) : (((sizeof(struct mm_freenode_s)) >> 4) & 0x1 ? 1 : 0))) : ((sizeof(struct mm_freenode_s)) & 0xc ? 2 + (((sizeof(struct mm_freenode_s)) >> 2) & 0x2 ? 1 + ((((sizeof(struct mm_freenode_s)) >> 2) >> 1) & 0x1 ? 1 : 0) : (((sizeof(struct mm_freenode_s)) >> 2) & 0x1 ? 1 : 0)) : ((sizeof(struct mm_freenode_s)) & 0x2 ? 1 + (((sizeof(struct mm_freenode_s)) >> 1) & 0x1 ? 1 : 0) : ((sizeof(struct mm_freenode_s)) & 0x1 ? 1 : 0))))))) - 1 : ((sizeof(struct mm_freenode_s)) & 0xffffffff00000000ul ? 32 + (((size_t)(sizeof(struct mm_freenode_s)) >> 32) & 0xffff0000 ? 16 + ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) & 0xff00 ? 8 + (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) & 0xf0 ? 4 + ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 4) & 0xc ? 2 + (((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 4) >> 2) & 0x2 ? 1 + ((((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 4) >> 2) >> 1) & 0x1 ? 1 : 0) : (((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 4) >> 2) & 0x1 ? 1 : 0)) : ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 4) & 0x2 ? 1 + (((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 4) >> 1) & 0x1 ? 1 : 0) : ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 4) & 0x1 ? 1 : 0))) : (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) & 0xc ? 2 + ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 2) & 0x2 ? 1 + (((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 2) >> 1) & 0x1 ? 1 : 0) : ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 2) & 0x1 ? 1 : 0)) : (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) & 0x2 ? 1 + ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) >> 1) & 0x1 ? 1 : 0) : (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 8) & 0x1 ? 1 : 0)))) : ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) & 0xf0 ? 4 + (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 4) & 0xc ? 2 + ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 4) >> 2) & 0x2 ? 1 + (((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 4) >> 2) >> 1) & 0x1 ? 1 : 0) : ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 4) >> 2) & 0x1 ? 1 : 0)) : (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 4) & 0x2 ? 1 + ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 4) >> 1) & 0x1 ? 1 : 0) : (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 4) & 0x1 ? 1 : 0))) : ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) & 0xc ? 2 + (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 2) & 0x2 ? 1 + ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 2) >> 1) & 0x1 ? 1 : 0) : (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 2) & 0x1 ? 1 : 0)) : ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) & 0x2 ? 1 + (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) >> 1) & 0x1 ? 1 : 0) : ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 16) & 0x1 ? 1 : 0))))) : (((size_t)(sizeof(struct mm_freenode_s)) >> 32) & 0xff00 ? 8 + ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) & 0xf0 ? 4 + (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 4) & 0xc ? 2 + ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 4) >> 2) & 0x2 ? 1 + (((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 4) >> 2) >> 1) & 0x1 ? 1 : 0) : ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 4) >> 2) & 0x1 ? 1 : 0)) : (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 4) & 0x2 ? 1 + ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 4) >> 1) & 0x1 ? 1 : 0) : (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 4) & 0x1 ? 1 : 0))) : ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) & 0xc ? 2 + (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 2) & 0x2 ? 1 + ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 2) >> 1) & 0x1 ? 1 : 0) : (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 2) & 0x1 ? 1 : 0)) : ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) & 0x2 ? 1 + (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) >> 1) & 0x1 ? 1 : 0) : ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 8) & 0x1 ? 1 : 0)))) : (((size_t)(sizeof(struct mm_freenode_s)) >> 32) & 0xf0 ? 4 + ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 4) & 0xc ? 2 + (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 4) >> 2) & 0x2 ? 1 + ((((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 4) >> 2) >> 1) & 0x1 ? 1 : 0) : (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 4) >> 2) & 0x1 ? 1 : 0)) : ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 4) & 0x2 ? 1 + (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 4) >> 1) & 0x1 ? 1 : 0) : ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 4) & 0x1 ? 1 : 0))) : (((size_t)(sizeof(struct mm_freenode_s)) >> 32) & 0xc ? 2 + ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 2) & 0x2 ? 1 + (((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 2) >> 1) & 0x1 ? 1 : 0) : ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 2) & 0x1 ? 1 : 0)) : (((size_t)(sizeof(struct mm_freenode_s)) >> 32) & 0x2 ? 1 + ((((size_t)(sizeof(struct mm_freenode_s)) >> 32) >> 1) & 0x1 ? 1 : 0) : (((size_t)(sizeof(struct mm_freenode_s)) >> 32) & 0x1 ? 1 : 0)))))) : ((sizeof(struct mm_freenode_s)) & 0xffff0000 ? 16 + (((sizeof(struct mm_freenode_s)) >> 16) & 0xff00 ? 8 + ((((sizeof(struct mm_freenode_s)) >> 16) >> 8) & 0xf0 ? 4 + (((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 4) & 0xc ? 2 + ((((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 4) >> 2) & 0x2 ? 1 + (((((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 4) >> 2) >> 1) & 0x1 ? 1 : 0) : ((((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 4) >> 2) & 0x1 ? 1 : 0)) : (((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 4) & 0x2 ? 1 + ((((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 4) >> 1) & 0x1 ? 1 : 0) : (((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 4) & 0x1 ? 1 : 0))) : ((((sizeof(struct mm_freenode_s)) >> 16) >> 8) & 0xc ? 2 + (((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 2) & 0x2 ? 1 + ((((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 2) >> 1) & 0x1 ? 1 : 0) : (((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 2) & 0x1 ? 1 : 0)) : ((((sizeof(struct mm_freenode_s)) >> 16) >> 8) & 0x2 ? 1 + (((((sizeof(struct mm_freenode_s)) >> 16) >> 8) >> 1) & 0x1 ? 1 : 0) : ((((sizeof(struct mm_freenode_s)) >> 16) >> 8) & 0x1 ? 1 : 0)))) : (((sizeof(struct mm_freenode_s)) >> 16) & 0xf0 ? 4 + ((((sizeof(struct mm_freenode_s)) >> 16) >> 4) & 0xc ? 2 + (((((sizeof(struct mm_freenode_s)) >> 16) >> 4) >> 2) & 0x2 ? 1 + ((((((sizeof(struct mm_freenode_s)) >> 16) >> 4) >> 2) >> 1) & 0x1 ? 1 : 0) : (((((sizeof(struct mm_freenode_s)) >> 16) >> 4) >> 2) & 0x1 ? 1 : 0)) : ((((sizeof(struct mm_freenode_s)) >> 16) >> 4) & 0x2 ? 1 + (((((sizeof(struct mm_freenode_s)) >> 16) >> 4) >> 1) & 0x1 ? 1 : 0) : ((((sizeof(struct mm_freenode_s)) >> 16) >> 4) & 0x1 ? 1 : 0))) : (((sizeof(struct mm_freenode_s)) >> 16) & 0xc ? 2 + ((((sizeof(struct mm_freenode_s)) >> 16) >> 2) & 0x2 ? 1 + (((((sizeof(struct mm_freenode_s)) >> 16) >> 2) >> 1) & 0x1 ? 1 : 0) : ((((sizeof(struct mm_freenode_s)) >> 16) >> 2) & 0x1 ? 1 : 0)) : (((sizeof(struct mm_freenode_s)) >> 16) & 0x2 ? 1 + ((((sizeof(struct mm_freenode_s)) >> 16) >> 1) & 0x1 ? 1 : 0) : (((sizeof(struct mm_freenode_s)) >> 16) & 0x1 ? 1 : 0))))) : ((sizeof(struct mm_freenode_s)) & 0xff00 ? 8 + (((sizeof(struct mm_freenode_s)) >> 8) & 0xf0 ? 4 + ((((sizeof(struct mm_freenode_s)) >> 8) >> 4) & 0xc ? 2 + (((((sizeof(struct mm_freenode_s)) >> 8) >> 4) >> 2) & 0x2 ? 1 + ((((((sizeof(struct mm_freenode_s)) >> 8) >> 4) >> 2) >> 1) & 0x1 ? 1 : 0) : (((((sizeof(struct mm_freenode_s)) >> 8) >> 4) >> 2) & 0x1 ? 1 : 0)) : ((((sizeof(struct mm_freenode_s)) >> 8) >> 4) & 0x2 ? 1 + (((((sizeof(struct mm_freenode_s)) >> 8) >> 4) >> 1) & 0x1 ? 1 : 0) : ((((sizeof(struct mm_freenode_s)) >> 8) >> 4) & 0x1 ? 1 : 0))) : (((sizeof(struct mm_freenode_s)) >> 8) & 0xc ? 2 + ((((sizeof(struct mm_freenode_s)) >> 8) >> 2) & 0x2 ? 1 + (((((sizeof(struct mm_freenode_s)) >> 8) >> 2) >> 1) & 0x1 ? 1 : 0) : ((((sizeof(struct mm_freenode_s)) >> 8) >> 2) & 0x1 ? 1 : 0)) : (((sizeof(struct mm_freenode_s)) >> 8) & 0x2 ? 1 + ((((sizeof(struct mm_freenode_s)) >> 8) >> 1) & 0x1 ? 1 : 0) : (((sizeof(struct mm_freenode_s)) >> 8) & 0x1 ? 1 : 0)))) : ((sizeof(struct mm_freenode_s)) & 0xf0 ? 4 + (((sizeof(struct mm_freenode_s)) >> 4) & 0xc ? 2 + ((((sizeof(struct mm_freenode_s)) >> 4) >> 2) & 0x2 ? 1 + (((((sizeof(struct mm_freenode_s)) >> 4) >> 2) >> 1) & 0x1 ? 1 : 0) : ((((sizeof(struct mm_freenode_s)) >> 4) >> 2) & 0x1 ? 1 : 0)) : (((sizeof(struct mm_freenode_s)) >> 4) & 0x2 ? 1 + ((((sizeof(struct mm_freenode_s)) >> 4) >> 1) & 0x1 ? 1 : 0) : (((sizeof(struct mm_freenode_s)) >> 4) & 0x1 ? 1 : 0))) : ((sizeof(struct mm_freenode_s)) & 0xc ? 2 + (((sizeof(struct mm_freenode_s)) >> 2) & 0x2 ? 1 + ((((sizeof(struct mm_freenode_s)) >> 2) >> 1) & 0x1 ? 1 : 0) : (((sizeof(struct mm_freenode_s)) >> 2) & 0x1 ? 1 : 0)) : ((sizeof(struct mm_freenode_s)) & 0x2 ? 1 + (((sizeof(struct mm_freenode_s)) >> 1) & 0x1 ? 1 : 0) : ((sizeof(struct mm_freenode_s)) & 0x1 ? 1 : 0))))))))):
at file: mm_heap/mm_mallinfo.c:69 task: /system/bin/init process: /system/bin/init 0xc000001a
[ 12.699000] up_dump_register: EPC: 000000008021462e
UPDATE 2: /system/bin/hello works OK on Ox64 and QEMU, but not hello. And free is still crashing on QEMU with CONFIG_BOARD_LATE_INITIALIZE=y: https://gist.github.com/lupyuen/9eb6526672fdeaadf04a57a87b7c1db9
NuttShell (NSH) NuttX-12.4.0
nsh> /system/bin/hello
Hello, World!!
nsh> hello
nsh: hello: command not found
nsh> uname -a
NuttX 12.4.0 9790248f9a-dirty Jun 24 2024 14:09:28 risc-v rv-virt
nsh> free
mm_foreach: region=0 node=0x80408d24 size=0 preceding=0 (A F)
mallinfo_handler: node=0x80408d24 size=0 preceding=0 (F)
_assert: Current Version: NuttX 12.4.0 9790248f9a Jun 24 2024 18:00:11 risc-v
_assert: Assertion failed : at file: mm_heap/mm_mallinfo.c:69 task: /system/bin/init process: /system/bin/init 0xc000001a
## Why is Node Size=0? Is the Heap corrupted? Or someone actually allocated Heap Size 0?
Wonder if this problem is happening because CONFIG_BOARD_LATE_INITIALIZE=y causes the Kernel Init to be performed on a Background Task (AppBringUp)? Which might be incompatible with this commit? Hmmm...
(FYI: Why is Ox64 using CONFIG_BOARD_LATE_INITIALIZE? That's because it needs to mount the Initial RAM Disk that contains the NuttX Apps. And mounting of File Systems won't work during the Early Kernel Init)
UPDATE 3: If we revert kmm_zalloc back to the original version, then free won't crash. Is the Late Init (AppBringUp) allocating the wrong kind of TCB, causing Heap Corruption? nuttx/sched/task/task_create.c
// creates and activates a new thread
int nxthread_create() {
/* Allocate a TCB for the new task. */
// We revert to:
tcb = kmm_zalloc(sizeof(struct task_tcb_s));
// Previously: tcb = kmm_zalloc(ttype == TCB_FLAG_TTYPE_KERNEL ?
// sizeof(struct tcb_s) : sizeof(struct task_tcb_s));
(Up Next: When CONFIG_BOARD_LATE_INITIALIZE=Y, why does QEMU knsh64 say that hello is not found? But /system/bin/hello is OK?)
UPDATE 4: Mainline QEMU knsh64 with CONFIG_BOARD_LATE_INITIALIZE=Y says "hello not found" because the Environment Variables are missing
## envp is missing!
nsh> hello
posix_spawn: path=hello, envp=0
nsh: hello: command not found
Rightfully, the Environment Variables should include PATH
## envp contains PATH
nsh> hello
posix_spawn: path=hello, envp=0x80200580
posix_spawn: envp[i]=PWD=/
posix_spawn: envp[i]=PATH=/system/bin
Hello, World!!
Why is the Environment missing with QEMU knsh64 and CONFIG_BOARD_LATE_INITIALIZE=Y? Is there a problem with tg_envp?
Hi @yf13 , this commit is also bricking most of the configurations on ESP32, ESP32-C3, ESP32-S3 and ESP32-C6 that enable Wi-Fi.
For instance, on esp32s3-devkitc:sta_softap:
make -j distclean
./tools/configure.sh esp32s3-devkit:sta_softap
make -j bootloader
make flash EXTRAFLAGS="-Wno-cpp -Werror" ESPTOOL_PORT=/dev/ttyUSB0 ESPTOOL_BINDIR=./ -s -j$(nproc)
minicom -D /dev/ttyUSB0
After setting the Wi-Fi credentials, it hangs:
nsh> wapi psk wlan0 mypassword 3
nsh> wapi essid wlan0 myssid 1
You can enable the wireless logs to see that the device hangs.
Can we revert it until all the issues are properly investigated?
Can we revert it until all the issues are properly investigated?
Yes I agree we should revert the commit for now. This is failing the Daily Test of Ox64 and Milk-V Duo S. And NuttX will be released any day now.
@yf13 We might need to ask the Mailing List for tips on troubleshooting our Kernel Heap Corruption, I see no easy way to track it down. Sorry I should have warned you that this is an extremely tricky PR, we will need plenty of Regression Testing. Thanks :-)