FreeRTOS-Kernel
FreeRTOS-Kernel copied to clipboard
FreeRTOS
[BUG] SMP mode - vTaskCoreAffinitySet does not yield
Describe the bug vTaskCoreAffinitySet does not yield when the modified task is in a ready state but not executing.
Scenario: FreeRTOS is running on 4 cores. Core 0 is executing:
for (size_t i = 0u; i < 4U; ++i)
{
// TaskHandle_t handle = xTaskCreateStaticAffinitySet(
// core_task, "core", MAIN_TASK_SIZE, NULL, MAIN_TASK_PRI, gCoreTaskStack[i], &gCoreTaskObj[i], (1 << i));
// configASSERT(handle != NULL);
TaskHandle_t handle = xTaskCreateStatic(
core_task, "core", MAIN_TASK_SIZE, NULL, MAIN_TASK_PRI, gCoreTaskStack[i], &gCoreTaskObj[i]);
configASSERT(handle != NULL);
vTaskCoreAffinitySet(handle, (1 << i));
}
No other tasks are running.
Outcome: Sometimes one task is not executed (5% times), but it is on ReadyList.
From the code instrumentation, I see that:
- when the core is executed then
if( taskTASK_IS_RUNNING( pxTCB ) == pdTRUE )was true - when the core is not executing then both
if( taskTASK_IS_RUNNING( pxTCB ) == pdTRUE )andif( ( uxPrevNotAllowedCores & uxCoreAffinityMask ) != 0U )were false and nobody yield assigned core.
It makes sense because:
- task was not running (it was in ready state)
- we did not added a new core
In our case, there was no other job for this core.
if( ( uxPrevNotAllowedCores & uxCoreAffinityMask ) != 0U ) is wrong, because it assumes that the task was blocked (the ready state was forgotten).
@WojciechJasko-BB
The tasks created by xTaskCreateStatic will be scheduled to run when the following conditions are met:
- The task's core affinity settings permit it to run on the core ( default no core affinity )
- The task has a higher priority than the currently running task on the core.
- configUSE_PREEMPTION is enabled.
Restricting the core affinity of a task in the ready state does not alter its state. This is because the task should be able to run already with it's original core affinity mask. Therefore, yield for the task is only required when new core is added to it's core affinity mask.
I use the following code for explanation:
for (size_t i = 0u; i < 4U; ++i)
{
/* By default, task is created with no core affinity. Task 0 is able to run on core 0, 1, 2, 3.
* So does task 1, 2 and 3. The only reason that the task is not running in this situation is that
* it doesn't have higher priority than any current running task.
* Let's assume that task 0, 1, 2 are running and task 3 is in ready state due to some higher priority
* task is running on core 3. */
TaskHandle_t handle = xTaskCreateStatic( core_task,
"core",
MAIN_TASK_SIZE,
NULL,
MAIN_TASK_PRI,
gCoreTaskStack[i],
&gCoreTaskObj[i]);
/* Setting core affinity of task 3 to core 3 only doesn't change it's state cause the same reason that
* core 3 is running a task with higher priority than task 3. Therefore, this task can't be preempted by task 3. */
vTaskCoreAffinitySet(handle, (1 << i));
}
We will need your help to provide the following information to investigate this problem.
- Is timer task enabled ( configUSE_TIMERS )? What is the priority of the timer task?
- What is the current running task on the core when the created task is in ready state?
@chinglee-iot Is timer task enabled ( configUSE_TIMERS )? What is the priority of the timer task? Yes, timers are enabled but not used. Timer task has piority configMAX_PRIORITIES - 1. It is the same priority as newly created task.
What is the current running task on the core when the created task is in ready state? There is no other task that can be run on that core.
There are 6 task started by user code (excluding timer task). 4x tasks that are assigned per core for counters incrementation. 1 task for printing counters values assigned to core 0. 1 task for serving uart assigned to core 0.
These 4 tasks are created after start of scheduler. Other 2 before.
In 5% cases, one counter is not incrementing. When I check it with debugger, it is on ready list. I also instrumented portYIELD_CORE and it is not called for specific core.
Thank you for your reply. From your information, it looks like that the task is in ready state and never get scheduled to run in 5% cases and the scheduler doesn't request the core to yield for this task.
I will need more information from you to investigate this problem.
Assuming that your task in ready state is supposed to run on core x. Can you help to provide pxCurrentTCBs[ x ] information with your debugger?
If you are using GDB, you can just print the *pxCurrentTCBs[ x ] to dump the fields in TCB_t.
Example dump:
(gdb) print *pxCurrentTCBs[0] $1 = {pxTopOfStack = 0x5555555e6390, uxCoreAffinityMask = 18446744073709551615, xStateListItem = {xItemValue = 0, pxNext = 0x5555555e64e0, pxPrevious = 0x555555596140 <pxReadyTasksLists+96>, pvOwner = 0x5555555e6280, pvContainer = 0x555555596130 <pxReadyTasksLists+80>}, xEventListItem = {xItemValue = 5, pxNext = 0x0, pxPrevious = 0x0, pvOwner = 0x5555555e6280, pvContainer = 0x0}, uxPriority = 2, pxStack = 0x5555555e6390, xTaskRunState = 0, uxTaskAttributes = 0, pcTaskName = "SMP Task\000\000\000", xPreemptionDisable = 0, pxEndOfStack = 0x5555555e64a4, uxCriticalNesting = 0, uxTCBNumber = 1, uxTaskNumber = 0, uxBasePriority = 2, uxMutexesHeld = 0, pxTaskTag = 0x0, ulNotifiedValue = {0, 0, 0, 0, 0}, ucNotifyState = "\000\000\000\000", ucDelayAborted = 0 '\000'}
@WojciechJasko-BB can you help us with further information as requested by @chinglee-iot
Hello, sorry for delay. In this run Cores 0, 2, 3 are running, but core 1 stucked in IDLE.
[1] struct tskTaskControlBlock * 0x0000000080FCE090 {pxTopOfStack=0x0000000080FCD460 {0},uxCoreAffinityMask=18446744073709551615,xStateListItem=... 0x0000000081010208
*([1]) struct tskTaskControlBlock {pxTopOfStack=0x0000000080FCD460 {0},uxCoreAffinityMask=18446744073709551615,xStateListItem=... 0x0000000080FCE090
pxTopOfStack unsigned long * 0x0000000080FCD460 {0} 0x0000000080FCE090
uxCoreAffinityMask unsigned long 18446744073709551615 0x0000000080FCE098
xStateListItem struct ListItem_t {xItemValue=0,pxNext=0x0000000080FCDFA0 {xItemValue=0,pxNext=0x0000000080FCDDA0 ...,pxPrevious=...,pxPrevious=... 0x0000000080FCE0A0
xItemValue unsigned int 0 0x0000000080FCE0A0
pxNext struct xLIST_ITEM * 0x0000000080FCDFA0 {xItemValue=0,pxNext=0x0000000080FCDDA0 {xItemValue=0,pxNext=...,pxPrevious=... 0x0000000080FCE0A8
*(pxNext) struct xLIST_ITEM {xItemValue=0,pxNext=0x0000000080FCDDA0 {xItemValue=0,pxNext=0x0000000080FCDEA0 ...,pxPrevious=...,pxPrevious=... 0x0000000080FCDFA0
xItemValue unsigned int 0 0x0000000080FCDFA0
pxNext struct xLIST_ITEM * 0x0000000080FCDDA0 {xItemValue=0,pxNext=0x0000000080FCDEA0 {xItemValue=0,pxNext=...,pxPrevious=... 0x0000000080FCDFA8
pxPrevious struct xLIST_ITEM * 0x0000000080FCE0A0 {xItemValue=0,pxNext=0x0000000080FCDFA0 {xItemValue=0,pxNext=...,pxPrevious=... 0x0000000080FCDFB0
pvOwner void * 0x0000000080FCDF90 0x0000000080FCDFB8
pvContainer struct xLIST * 0x0000000080044B78 {uxNumberOfItems=4,pxIndex=0x0000000080044B88 {xItemValue=4294967295,pxNext=...,xListEnd=... 0x0000000080FCDFC0
pxPrevious struct xLIST_ITEM * 0x0000000080044B88 {xItemValue=4294967295,pxNext=0x0000000080FCE0A0 {xItemValue=...,pxPrevious=... 0x0000000080FCE0B0
*(pxPrevious) struct xLIST_ITEM {xItemValue=4294967295,pxNext=0x0000000080FCE0A0 {xItemValue=0,pxNext=0x0000000080FCDFA0 ...,pxPrevious=...,pxPrevious=... 0x0000000080044B88
xItemValue unsigned int 4294967295 0x0000000080044B88
pxNext struct xLIST_ITEM * 0x0000000080FCE0A0 {xItemValue=0,pxNext=0x0000000080FCDFA0 {xItemValue=0,pxNext=...,pxPrevious=... 0x0000000080044B90
pxPrevious struct xLIST_ITEM * 0x0000000080FCDEA0 {xItemValue=0,pxNext=0x0000000080044B88 {xItemValue=4294967295,pxNext=...,pxPrevious=... 0x0000000080044B98
pvOwner void * 0x0000000000000000 0x0000000080044BA0
pvContainer struct xLIST * 0x0000000080044BB0 {uxNumberOfItems=4294967295,pxIndex=0x0000000080044BB0 {xItemValue=...,xListEnd=... 0x0000000080044BA8
pvOwner void * 0x0000000080FCE090 0x0000000080FCE0B8
pvContainer struct xLIST * 0x0000000080044B78 {uxNumberOfItems=4,pxIndex=0x0000000080044B88 {xItemValue=4294967295,pxNext=...,xListEnd=... 0x0000000080FCE0C0
*(pvContainer) struct xLIST {uxNumberOfItems=4,pxIndex=0x0000000080044B88 {xItemValue=4294967295,pxNext=0x0000000080FCE0A0 ...,pxPrevious=...,xListEnd=... 0x0000000080044B78
uxNumberOfItems unsigned long 4 0x0000000080044B78
pxIndex struct xLIST_ITEM * 0x0000000080044B88 {xItemValue=4294967295,pxNext=0x0000000080FCE0A0 {xItemValue=...,pxPrevious=... 0x0000000080044B80
xListEnd struct MiniListItem_t {xItemValue=4294967295,pxNext=0x0000000080FCE0A0 {xItemValue=0,pxNext=0x0000000080FCDFA0 ...,pxPrevious=...,pxPrevious=... 0x0000000080044B88
xEventListItem struct ListItem_t {xItemValue=32,pxNext=0x0000000000000000 {xItemValue=0,pxNext=0x0000000000000000 ...,pxPrevious=...,pxPrevious=... 0x0000000080FCE0C8
xItemValue unsigned int 32 0x0000000080FCE0C8
pxNext struct xLIST_ITEM * 0x0000000000000000 {xItemValue=0,pxNext=0x0000000000000000 {xItemValue=0,pxNext=...,pxPrevious=... 0x0000000080FCE0D0
pxPrevious struct xLIST_ITEM * 0x0000000000000000 {xItemValue=0,pxNext=0x0000000000000000 {xItemValue=0,pxNext=...,pxPrevious=... 0x0000000080FCE0D8
pvOwner void * 0x0000000080FCE090 0x0000000080FCE0E0
pvContainer struct xLIST * 0x0000000000000000 {uxNumberOfItems=0,pxIndex=0x0000000000000000 {xItemValue=0,pxNext=...,xListEnd=... 0x0000000080FCE0E8
uxPriority unsigned long 0 0x0000000080FCE0F0
pxStack unsigned long * 0x0000000080FCB590 {11936128518282651045} 0x0000000080FCE0F8
xTaskRunState long 1 0x0000000080FCE100
uxTaskAttributes unsigned long 1 0x0000000080FCE108
pcTaskName unsigned char[32] [73 'I',68 'D',76 'L',69 'E',51 '3'...] 0x0000000080FCE110
uxCriticalNesting unsigned long 0 0x0000000080FCE130
uxTCBNumber unsigned long 5 0x0000000080FCE138
uxTaskNumber unsigned long 0 0x0000000080FCE140
uxBasePriority unsigned long 0 0x0000000080FCE148
uxMutexesHeld unsigned long 0 0x0000000080FCE150
pxTaskTag long (void *) 0x0000000000000000 0x0000000080FCE158
pvThreadLocalStoragePointers void *[4] [0x0000000000000000,0x0000000000000000,0x0000000000000000,0x0000000000000000] 0x0000000080FCE160
ulRunTimeCounter unsigned int 1680 0x0000000080FCE180
ulNotifiedValue unsigned int[1] [0] 0x0000000080FCE184
ucNotifyState unsigned char[1] [0 '\x00'] 0x0000000080FCE188
ucStaticallyAllocated unsigned char 2 '\x02' 0x0000000080FCE189
iTaskErrno int 0 0x0000000080FCE18C
@WojciechJasko-BB
Thanks for your information. We want to know if calling the vTaskCoreAffinitySet is the cause to this problem. Can you help to run you code again without setting core affinity and share us the result?
for (size_t i = 0u; i < 4U; ++i)
{
// TaskHandle_t handle = xTaskCreateStaticAffinitySet(
// core_task, "core", MAIN_TASK_SIZE, NULL, MAIN_TASK_PRI, gCoreTaskStack[i], &gCoreTaskObj[i], (1 << i));
// configASSERT(handle != NULL);
TaskHandle_t handle = xTaskCreateStatic(
core_task, "core", MAIN_TASK_SIZE, NULL, MAIN_TASK_PRI, gCoreTaskStack[i], &gCoreTaskObj[i]);
configASSERT(handle != NULL);
/* Comment out this line of code and run the application again. */
// vTaskCoreAffinitySet(handle, (1 << i));
}
Yes it works. What is more, using the API to create task and set it's affinity at once works as well.
@WojciechJasko-BB Can you help me double confirm your experiment result?
( x ) Sometimes one task is not executed (5% times), but it is on ReadyList.
for (size_t i = 0u; i < 4U; ++i)
{
// TaskHandle_t handle = xTaskCreateStaticAffinitySet(
// core_task, "core", MAIN_TASK_SIZE, NULL, MAIN_TASK_PRI, gCoreTaskStack[i], &gCoreTaskObj[i], (1 << i));
// configASSERT(handle != NULL);
TaskHandle_t handle = xTaskCreateStatic(
core_task, "core", MAIN_TASK_SIZE, NULL, MAIN_TASK_PRI, gCoreTaskStack[i], &gCoreTaskObj[i]);
configASSERT(handle != NULL);
vTaskCoreAffinitySet(handle, (1 << i));
}
It works well If removing the condition check in vTaskCoreAffinitySet.
void vTaskCoreAffinitySet( const TaskHandle_t xTask,
UBaseType_t uxCoreAffinityMask )
{
....
/* Comment out the following condition. The application works well.
// if( ( uxPrevNotAllowedCores & uxCoreAffinityMask ) != 0U )
{
prvYieldForTask( pxTCB );
}
}
( O ) No core affinity. It works well. All the counters are increased by the tasks. ( edit to fix code example )
for (size_t i = 0u; i < 4U; ++i)
{
// TaskHandle_t handle = xTaskCreateStaticAffinitySet(
// core_task, "core", MAIN_TASK_SIZE, NULL, MAIN_TASK_PRI, gCoreTaskStack[i], &gCoreTaskObj[i], (1 << i));
// configASSERT(handle != NULL);
TaskHandle_t handle = xTaskCreateStatic(
core_task, "core", MAIN_TASK_SIZE, NULL, MAIN_TASK_PRI, gCoreTaskStack[i], &gCoreTaskObj[i]);
configASSERT(handle != NULL);
/* Comment out this line of code and run the application again. */
// vTaskCoreAffinitySet(handle, (1 << i));
}
( O ) Use xTaskCreateStaticAffinitySet to set core affinity when it is created. It works well. All the counters are increased by the tasks.
for (size_t i = 0u; i < 4U; ++i)
{
TaskHandle_t handle = xTaskCreateStaticAffinitySet(
core_task, "core", MAIN_TASK_SIZE, NULL, MAIN_TASK_PRI, gCoreTaskStack[i], &gCoreTaskObj[i], (1 << i));
// configASSERT(handle != NULL);
// TaskHandle_t handle = xTaskCreateStatic(
// core_task, "core", MAIN_TASK_SIZE, NULL, MAIN_TASK_PRI, gCoreTaskStack[i], &gCoreTaskObj[i]);
configASSERT(handle != NULL);
// vTaskCoreAffinitySet(handle, (1 << i));
}
@WojciechJasko-BB I am able to reproduce this problem in unit test. In summary, I think your observation about the problem is correct. This problem happens when a core is not able to handle a yield request in time. Handling tasks of ready state in the vTaskCoreAffinitySet implementation can solve the problem. We will create a pull request to fix this issue and update in this issue.
I also share my observation here. We can simplify the problem with creating one task only. In the vMainTask function, a task is created and set the core affinity.
void vMainTask( void * pvParameters )
{
TaskHandle_t handle = xTaskCreateStatic(
core_task, "core", MAIN_TASK_SIZE, NULL, MAIN_TASK_PRI, gCoreTaskStack, &gCoreTaskObj);
vTaskCoreAffinitySet(handle, (1 << 1 ) );
}
Considering the following execute sequence:
- The system has 4 cores and main task is the highest priority task running on core 0. The other cores are running an idle task.
- main task calls
xTaskCreateStaticto create the core task and request core 3 to yield for core task - core 3 is not able to handle the yield request in time so the core task is still in ready state.
- main task continue to calls
vTaskCoreAffinitySetto change core task core affinity to core 1. - The core task core affinity is limited to core 1. The core task is still in ready state due to step 3 reason. In the current implementation,
prvYieldForTaskis not called for the core task when limiting core affinity of a blocking task, which result in core 1 is not requested to select core task to run. - core 3 handles the yield request now. Previously, core 3 was supposed to select the core task to run, but due to core affinity being limited to Core 1 only now, core 3 could only select an idle task to run.
The result is that the core task has higher priority than idle task but can't run on core 1, which is probably the situation you observed. This problem is not observed if core 3 is able to handle the yield request before core 1 calls vTaskCoreAffinitySet.
Thank you again for reporting this issue and your patience to help us understanding this problem.
I am closing this issue as the PR fixing this is merged. Please feel free to re-open if you still face some issue.
