Workflow Task Timeout not always respected

[Sushisource]

Expected Behavior

When setting https://github.com/temporalio/api/blob/master/temporal/api/workflowservice/v1/request_response.proto#L158 I expect workflow tasks to always time out with the specified duration.

Actual Behavior

Sometimes the timeout may not be respected. In particular, if the task is being processed on a sticky queue. I have attached a history where the execution started event clearly shows a 1s task timeout, but there is a task timeout that doesn't happen for 10 seconds: a20228ec-d12b-43dc-91e7-f00350e3f2dc_events.json

Steps to Reproduce the Problem

I want to say that I've seen this work properly on sticky queues before, but in my case I had to explicitly call reset on the sticky queue to get things to proceed quickly. In my test, there is an outstanding workflow update while the task times out. Perhaps it's a contributing factor.

This diff can be applied to the test added here to repro

diff --git a/tests/integ_tests/update_tests.rs b/tests/integ_tests/update_tests.rs
index 26254d40e..9eea536df 100644
--- a/tests/integ_tests/update_tests.rs
+++ b/tests/integ_tests/update_tests.rs
@@ -814,6 +814,7 @@ async fn task_failure_after_update() {
 async fn worker_restarted_in_middle_of_update() {
     let wf_name = "worker_restarted_in_middle_of_update";
     let mut starter = CoreWfStarter::new(wf_name);
+    starter.workflow_options.task_timeout = Some(Duration::from_secs(1));
     let mut worker = starter.worker().await;
     let client = starter.get_client().await;

@@ -886,10 +887,10 @@ async fn worker_restarted_in_middle_of_update() {
         // Allow it to start again, the second time
         BARR.wait().await;
         // Poke the workflow off the sticky queue to get it to complete faster than WFT timeout
-        client
-            .reset_sticky_task_queue(wf_id.clone(), run_id.clone())
-            .await
-            .unwrap();
+        // client
+        //     .reset_sticky_task_queue(wf_id.clone(), run_id.clone())
+        //     .await
+        //     .unwrap();
     };
     let run = async {
         // This run attempt will get shut down

Specifications

• Version:
• Platform: