cadence
cadence copied to clipboard
Published
20 hours ago •
uber
uber
cron workflow does not run as expected
I created a new workflow, CronSchedule is * * * * *, but occasionally it takes a few minutes to execute。
time endTime_include beginTime_exclude workflowId runId
08-18 06:24:00 2021-08-18 06:24:00.211 2021-08-18 06:23:01.683 cron1_0828295b-1330-44c0-9d2c-fcf50e96f634 7e716dca-5ac4-4c91-b777-d7fd300ddb95
08-18 06:27:01 2021-08-18 06:27:01.651 0001-01-01 00:00:00.000 cron1_0828295b-1330-44c0-9d2c-fcf50e96f634 316e068d-6bb4-43f7-b70a-dacdb8024a0b
the history of 7e716dca-5ac4-4c91-b777-d7fd300ddb95 is
1 WorkflowExecutionStarted {WorkflowType:{Name:sampleCronWorkflow}, TaskList:{Name:cronGroup},
Input:[], ExecutionStartToCloseTimeoutSeconds:60,
TaskStartToCloseTimeoutSeconds:60,
ContinuedExecutionRunId:e56be2b1-cb7b-4979-8f97-436cd3a7a07c,
Initiator:CronSchedule, ContinuedFailureDetails:[],
LastCompletionResult:[{"EndTime":"2021-08-18T06:23:01.683795225Z"}],
OriginalExecutionRunId:7e716dca-5ac4-4c91-b777-d7fd300ddb95,
FirstExecutionRunId:3f22e2ec-5901-4d93-bb0f-52a5dfa746e1, Attempt:0,
CronSchedule:* * * * *, FirstDecisionTaskBackoffSeconds:58,
PrevAutoResetPoints:{Points:[len=2]}, Header:{Fields:map{}}}
2 DecisionTaskScheduled {TaskList:{Name:cronGroup},
StartToCloseTimeoutSeconds:60,
Attempt:0}
3 DecisionTaskStarted {ScheduledEventId:2,
Identity:28575@xxxxxxx@cronGroup,
RequestId:7eda403a-8893-485a-b7e2-5a783f708183}
4 DecisionTaskCompleted {ExecutionContext:[],
ScheduledEventId:2, StartedEventId:3,
Identity:28575@xxxxxxx@cronGroup,
BinaryChecksum:1fc4cac14392e74c19db8a0e1ca32b5f}
5 ActivityTaskScheduled {ActivityId:0,
ActivityType:{Name:mwork/cronwork.sampleCronActivity},
TaskList:{Name:cronGroup},
Input:["2021-08-18T06:23:01.683795225Z"
"2021-08-18T06:24:00.211491248Z"],
ScheduleToCloseTimeoutSeconds:60,
ScheduleToStartTimeoutSeconds:60,
StartToCloseTimeoutSeconds:60,
HeartbeatTimeoutSeconds:0,
DecisionTaskCompletedEventId:4, Header:{Fields:map{}}}
6 ActivityTaskStarted {ScheduledEventId:5,
Identity:28575@xxxxxxx@cronGroup,
RequestId:7390e913-7e46-4e0d-98db-f800f5fda43c,
Attempt:0, LastFailureDetails:[]}
7 ActivityTaskCompleted {Result:[], ScheduledEventId:5, StartedEventId:6,
Identity:28575@xxxxxxx@cronGroup}
8 DecisionTaskScheduled {TaskList:{Name:xxxxxxx:96fe1455-6524-4bdd-a7ac-061ec739f090},
StartToCloseTimeoutSeconds:60, Attempt:0}
9 DecisionTaskStarted {ScheduledEventId:8,
Identity:28575@xxxxxxx@cronGroup,
RequestId:4d3ad253-6703-492e-b87e-95b5b670ce90}
10 DecisionTaskCompleted {ExecutionContext:[],
ScheduledEventId:8, StartedEventId:9,
Identity:28575@xxxxxxx@cronGroup,
BinaryChecksum:1fc4cac14392e74c19db8a0e1ca32b5f}
11 WorkflowExecutionContinuedAsNew {NewExecutionRunId:1580e056-522e-4512-92a9-04c748713f7f,
WorkflowType:{Name:sampleCronWorkflow}, TaskList:{Name:cronGroup},
Input:[], ExecutionStartToCloseTimeoutSeconds:60,
TaskStartToCloseTimeoutSeconds:60,
DecisionTaskCompletedEventId:10, BackoffStartIntervalInSeconds:60,
Initiator:CronSchedule, FailureDetails:[],
LastCompletionResult:[{"EndTime":"2021-08-18T06:24:00.211491248Z"}],
Header:{Fields:map{}}}
the history of 316e068d-6bb4-43f7-b70a-dacdb8024a0b is ,
1 WorkflowExecutionStarted {WorkflowType:{Name:sampleCronWorkflow},
TaskList:{Name:cronGroup}, Input:[],
ExecutionStartToCloseTimeoutSeconds:60,
TaskStartToCloseTimeoutSeconds:60,
ContinuedExecutionRunId:1580e056-522e-4512-92a9-04c748713f7f,
Initiator:CronSchedule,
ContinuedFailureReason:cadenceInternal:Timeout
START_TO_CLOSE, ContinuedFailureDetails:[],
LastCompletionResult:[],
OriginalExecutionRunId:316e068d-6bb4-43f7-b70a-dacdb8024a0b,
FirstExecutionRunId:3f22e2ec-5901-4d93-bb0f-52a5dfa746e1,
Attempt:0, CronSchedule:* * * * *,
FirstDecisionTaskBackoffSeconds:60,
PrevAutoResetPoints:{Points:[len=2]}, Header:{Fields:map{}}}
2 DecisionTaskScheduled {TaskList:{Name:cronGroup},
StartToCloseTimeoutSeconds:60,
Attempt:0}
3 DecisionTaskStarted {ScheduledEventId:2,
Identity:28575@xxxxxxx@cronGroup,
RequestId:5d013f60-1658-42af-b4b3-f59f4895c22a}
4 DecisionTaskCompleted {ExecutionContext:[],
ScheduledEventId:2, StartedEventId:3,
Identity:28575@xxxxxxx@cronGroup,
BinaryChecksum:1fc4cac14392e74c19db8a0e1ca32b5f}
5 ActivityTaskScheduled {ActivityId:0,
ActivityType:{Name:mwork/cronwork.sampleCronActivity},
TaskList:{Name:cronGroup},
Input:["0001-01-01T00:00:00Z"
"2021-08-18T06:27:01.651795985Z"],
ScheduleToCloseTimeoutSeconds:60,
ScheduleToStartTimeoutSeconds:60,
StartToCloseTimeoutSeconds:60,
HeartbeatTimeoutSeconds:0,
DecisionTaskCompletedEventId:4, Header:{Fields:map{}}}
6 ActivityTaskStarted {ScheduledEventId:5,
Identity:28575@xxxxxxx@cronGroup,
RequestId:667e7857-01ab-448f-ab8a-a62551cf93b6,
Attempt:0, LastFailureDetails:[]}
7 ActivityTaskCompleted {Result:[], ScheduledEventId:5, StartedEventId:6,
Identity:28575@xxxxxxx@cronGroup}
8 DecisionTaskScheduled {TaskList:{Name:xxxxxxx:96fe1455-6524-4bdd-a7ac-061ec739f090},
StartToCloseTimeoutSeconds:60, Attempt:0}
9 DecisionTaskStarted {ScheduledEventId:8,
Identity:28575@xxxxxxx@cronGroup,
RequestId:d0fd9151-8566-4af8-8d8c-6b382f356ef1}
10 DecisionTaskCompleted {ExecutionContext:[],
ScheduledEventId:8, StartedEventId:9,
Identity:28575@xxxxxxx@cronGroup,
BinaryChecksum:1fc4cac14392e74c19db8a0e1ca32b5f}
11 WorkflowExecutionContinuedAsNew {NewExecutionRunId:1447674e-77bb-48ca-b142-8327e62b323b,
WorkflowType:{Name:sampleCronWorkflow}, TaskList:{Name:cronGroup},
Input:[], ExecutionStartToCloseTimeoutSeconds:60,
TaskStartToCloseTimeoutSeconds:60,
DecisionTaskCompletedEventId:10, BackoffStartIntervalInSeconds:58,
Initiator:CronSchedule, FailureDetails:[],
LastCompletionResult:[{"EndTime":"2021-08-18T06:27:01.651795985Z"}],
Header:{Fields:map{}}}
The worker did not execute the task whose runid is 1580e056-522e-4512-92a9-04c748713f7f， the 1580e056-522e-4512-92a9-04c748713f7f history is
1 WorkflowExecutionStarted {WorkflowType:{Name:sampleCronWorkflow}, TaskList:{Name:cronGroup},
Input:[], ExecutionStartToCloseTimeoutSeconds:60,
TaskStartToCloseTimeoutSeconds:60,
ContinuedExecutionRunId:7e716dca-5ac4-4c91-b777-d7fd300ddb95,
Initiator:CronSchedule, ContinuedFailureDetails:[],
LastCompletionResult:[{"EndTime":"2021-08-18T06:24:00.211491248Z"}],
OriginalExecutionRunId:1580e056-522e-4512-92a9-04c748713f7f,
FirstExecutionRunId:3f22e2ec-5901-4d93-bb0f-52a5dfa746e1, Attempt:0,
CronSchedule:* * * * *, FirstDecisionTaskBackoffSeconds:60,
PrevAutoResetPoints:{Points:[len=2]}, Header:{Fields:map{}}}
2 DecisionTaskScheduled {TaskList:{Name:cronGroup},
StartToCloseTimeoutSeconds:60,
Attempt:0}
3 DecisionTaskStarted {ScheduledEventId:2,
Identity:28575@xxxxxxx@cronGroup,
RequestId:f5354f8f-7c83-4cd9-8b92-4cdca1475f9e}
4 DecisionTaskCompleted {ExecutionContext:[],
ScheduledEventId:2, StartedEventId:3,
Identity:28575@xxxxxxx@cronGroup,
BinaryChecksum:1fc4cac14392e74c19db8a0e1ca32b5f}
5 ActivityTaskScheduled {ActivityId:0,
ActivityType:{Name:mwork/cronwork.sampleCronActivity},
TaskList:{Name:cronGroup},
Input:["2021-08-18T06:24:00.211491248Z"
"2021-08-18T06:25:03.269078017Z"],
ScheduleToCloseTimeoutSeconds:60,
ScheduleToStartTimeoutSeconds:60,
StartToCloseTimeoutSeconds:60,
HeartbeatTimeoutSeconds:0,
DecisionTaskCompletedEventId:4, Header:{Fields:map{}}}
6 WorkflowExecutionContinuedAsNew {NewExecutionRunId:316e068d-6bb4-43f7-b70a-dacdb8024a0b,
WorkflowType:{Name:sampleCronWorkflow},
TaskList:{Name:cronGroup}, Input:[],
ExecutionStartToCloseTimeoutSeconds:60,
TaskStartToCloseTimeoutSeconds:60,
DecisionTaskCompletedEventId:-23,
BackoffStartIntervalInSeconds:60,
Initiator:CronSchedule,
FailureReason:cadenceInternal:Timeout START_TO_CLOSE,
FailureDetails:[], LastCompletionResult:[],
Header:{Fields:map{}}}
History Source: Default Storage
============Workflow Pending activities============
[
{
"ActivityID": "0",
"ActivityType": {
"name": "mwork/cronwork.sampleCronActivity"
},
"State": "SCHEDULED",
"ScheduledTimestamp": "2021-08-18T14:25:03+08:00"
}
]
I just used cadence and I can’t see where the problem is. Ask for help
Version
- v0.21.3
One common mistake is that’s because the workflow timeout is more than cron schedule.
If previous execution hasn’t closed, it would delay the next run for cron workflow. This is to ensure the ID uniqueness guarantee and also the workflow timeout guarantee.
If you wish to run the cron every minute, the workflow timeout must be less than or equal to a minute
