julep
julep copied to clipboard
Published
20 hours ago •
julep-ai
julep-ai
[Bug]: `DeadlockError` on `api_call` tool-calling with large response payloads
trafficstars
📜 Description
Worker Stack Trace:
worker-1 | INFO:temporalio.workflow:TaskExecutionWorkflow for task cf6e3294-66de-4aba-b019-bd3733d6cd7c [LOC main.0] ({'attempt': 1, 'namespace': 'default', 'run_id': 'd8fcd064-e206-480a-907a-42c08d41dec3', 'task_queue': 'julep-task-queue', 'workflow_id': '067c1696-664c-726a-8000-d6a82280fc51', 'workflow_type': 'TaskExecutionWorkflow'})
worker-1 | INFO:temporalio.workflow:Executing step 0 of type ToolCallStep ({'attempt': 1, 'namespace': 'default', 'run_id': 'd8fcd064-e206-480a-907a-42c08d41dec3', 'task_queue': 'julep-task-queue', 'workflow_id': '067c1696-664c-726a-8000-d6a82280fc51', 'workflow_type': 'TaskExecutionWorkflow'})
worker-1 | INFO:temporalio.workflow:Processing outcome for step 0 ({'attempt': 1, 'namespace': 'default', 'run_id': 'd8fcd064-e206-480a-907a-42c08d41dec3', 'task_queue': 'julep-task-queue', 'workflow_id': '067c1696-664c-726a-8000-d6a82280fc51', 'workflow_type': 'TaskExecutionWorkflow'})
worker-1 | INFO:httpx:HTTP Request: GET https://www.tirabeauty.com/ext/plpoffers/application/api/v1.0/collections/skin/items?filters=false&page_id=1&page_size=1000 "HTTP/1.1 200 OK"
worker-1 | INFO:temporalio.workflow:Transitioning after step 0 ({'attempt': 1, 'namespace': 'default', 'run_id': 'd8fcd064-e206-480a-907a-42c08d41dec3', 'task_queue': 'julep-task-queue', 'workflow_id': '067c1696-664c-726a-8000-d6a82280fc51', 'workflow_type': 'TaskExecutionWorkflow'})
worker-1 | ERROR:temporalio.worker._workflow:Failed handling activation on workflow with run ID d8fcd064-e206-480a-907a-42c08d41dec3
worker-1 | Traceback (most recent call last):
worker-1 | File "/app/agents_api/common/interceptors.py", line 94, in <genexpr>
worker-1 | size += sum(get_deep_size(item, seen) for item in obj)
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 94, in get_deep_size
worker-1 | size += sum(get_deep_size(item, seen) for item in obj)
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 97, in <genexpr>
worker-1 | size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 97, in get_deep_size
worker-1 | size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 97, in <genexpr>
worker-1 | size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 97, in get_deep_size
worker-1 | size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 97, in <genexpr>
worker-1 | size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 97, in get_deep_size
worker-1 | size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 97, in <genexpr>
worker-1 | size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 97, in get_deep_size
worker-1 | size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 94, in <genexpr>
worker-1 | size += sum(get_deep_size(item, seen) for item in obj)
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 94, in get_deep_size
worker-1 | size += sum(get_deep_size(item, seen) for item in obj)
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 97, in <genexpr>
worker-1 | size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 97, in get_deep_size
worker-1 | size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 97, in <genexpr>
worker-1 | size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 97, in get_deep_size
worker-1 | size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 97, in <genexpr>
worker-1 | size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 97, in get_deep_size
worker-1 | size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 101, in get_deep_size
worker-1 | size += get_deep_size(obj.__dict__, seen)
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 115, in is_too_large
worker-1 | return get_deep_size(result) > blob_store_cutoff_kb * 1024
worker-1 | ^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 150, in offload_if_large
worker-1 | if is_too_large(result):
worker-1 | ^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 316, in start_activity
worker-1 | input.args = [offload_if_large(arg) for arg in input.args]
worker-1 | ^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py", line 1180, in workflow_start_activity
worker-1 | return self._outbound.start_activity(
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/.venv/lib/python3.12/site-packages/temporalio/workflow.py", line 2053, in execute_activity
worker-1 | return await _Runtime.current().workflow_start_activity(
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/workflows/task_execution/transition.py", line 54, in transition
worker-1 | await workflow.execute_activity(
worker-1 | File "/app/agents_api/workflows/task_execution/__init__.py", line 811, in run
worker-1 | final_state = await transition(
worker-1 | ^^^^^^^^^^^^^^^^^
worker-1 | File "/app/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py", line 2140, in execute_workflow
worker-1 | return await input.run_fn(*args)
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/.venv/lib/python3.12/site-packages/temporalio/worker/_interceptor.py", line 328, in execute_workflow
worker-1 | return await self.next.execute_workflow(input)
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 216, in handle_execution_with_errors
worker-1 | return await execution_fn(input)
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 303, in execute_workflow
worker-1 | return await handle_execution_with_errors(
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/agents_api/common/interceptors.py", line 171, in wrapper
worker-1 | result = await func(self, input)
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/.venv/lib/python3.12/site-packages/temporalio/worker/_interceptor.py", line 328, in execute_workflow
worker-1 | return await self.next.execute_workflow(input)
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/.venv/lib/python3.12/site-packages/temporalio/contrib/opentelemetry.py", line 356, in execute_workflow
worker-1 | return await super().execute_workflow(input)
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py", line 857, in run_workflow
worker-1 | result = await self._inbound.execute_workflow(input)
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py", line 1808, in _run_top_level_workflow_function
worker-1 | await coro
worker-1 | File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run
worker-1 | self._context.run(self._callback, *self._args)
worker-1 | File "/app/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py", line 1785, in _run_once
worker-1 | handle._run()
worker-1 | File "/app/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py", line 377, in activate
worker-1 | self._run_once(check_conditions=index == 1 or index == 2)
worker-1 | File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 59, in run
worker-1 | result = self.fn(*self.args, **self.kwargs)
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 93, in _worker
worker-1 | work_item.run()
worker-1 | File "/usr/local/lib/python3.12/threading.py", line 1012, in run
worker-1 | self._target(*self._args, **self._kwargs)
worker-1 | File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
worker-1 | self.run()
worker-1 | File "/usr/local/lib/python3.12/threading.py", line 1032, in _bootstrap
worker-1 | self._bootstrap_inner()
worker-1 | temporalio.worker._workflow._DeadlockError: [TMPRL1101] Potential deadlock detected: workflow didn't yield within 2 second(s).
worker-1 | 2025-02-28T07:45:06.846949Z WARN temporal_sdk_core::worker::workflow: Failing workflow task run_id=d8fcd064-e206-480a-907a-42c08d41dec3 failure=Failure { failure: Some(Failure { message: "[TMPRL1101] Potential deadlock detected: workflow didn't yield within 2 second(s).", source: "", stack_trace: " File \"/app/agents_api/common/interceptors.py\", line 94, in <genexpr>\n size += sum(get_deep_size(item, seen) for item in obj)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 94, in get_deep_size\n size += sum(get_deep_size(item, seen) for item in obj)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 97, in <genexpr>\n size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())\n ^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 97, in get_deep_size\n size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 97, in <genexpr>\n size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())\n ^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 97, in get_deep_size\n size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 97, in <genexpr>\n size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())\n ^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 97, in get_deep_size\n size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 97, in <genexpr>\n size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())\n ^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 97, in get_deep_size\n size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 94, in <genexpr>\n size += sum(get_deep_size(item, seen) for item in obj)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 94, in get_deep_size\n size += sum(get_deep_size(item, seen) for item in obj)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 97, in <genexpr>\n size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())\n ^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 97, in get_deep_size\n size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 97, in <genexpr>\n size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())\n ^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 97, in get_deep_size\n size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 97, in <genexpr>\n size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())\n ^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 97, in get_deep_size\n size += sum(get_deep_size(k, seen) + get_deep_size(v, seen) for k, v in obj.items())\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 101, in get_deep_size\n size += get_deep_size(obj.__dict__, seen)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 115, in is_too_large\n return get_deep_size(result) > blob_store_cutoff_kb * 1024\n ^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 150, in offload_if_large\n if is_too_large(result):\n ^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 316, in start_activity\n input.args = [offload_if_large(arg) for arg in input.args]\n ^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py\", line 1180, in workflow_start_activity\n return self._outbound.start_activity(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/.venv/lib/python3.12/site-packages/temporalio/workflow.py\", line 2053, in execute_activity\n return await _Runtime.current().workflow_start_activity(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/workflows/task_execution/transition.py\", line 54, in transition\n await workflow.execute_activity(\n\n File \"/app/agents_api/workflows/task_execution/__init__.py\", line 811, in run\n final_state = await transition(\n ^^^^^^^^^^^^^^^^^\n\n File \"/app/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py\", line 2140, in execute_workflow\n return await input.run_fn(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/.venv/lib/python3.12/site-packages/temporalio/worker/_interceptor.py\", line 328, in execute_workflow\n return await self.next.execute_workflow(input)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 216, in handle_execution_with_errors\n return await execution_fn(input)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 303, in execute_workflow\n return await handle_execution_with_errors(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/agents_api/common/interceptors.py\", line 171, in wrapper\n result = await func(self, input)\n ^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/.venv/lib/python3.12/site-packages/temporalio/worker/_interceptor.py\", line 328, in execute_workflow\n return await self.next.execute_workflow(input)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/.venv/lib/python3.12/site-packages/temporalio/contrib/opentelemetry.py\", line 356, in execute_workflow\n return await super().execute_workflow(input)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py\", line 857, in run_workflow\n result = await self._inbound.execute_workflow(input)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/app/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py\", line 1808, in _run_top_level_workflow_function\n await coro\n\n File \"/usr/local/lib/python3.12/asyncio/events.py\", line 88, in _run\n self._context.run(self._callback, *self._args)\n\n File \"/app/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py\", line 1785, in _run_once\n handle._run()\n\n File \"/app/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py\", line 377, in activate\n self._run_once(check_conditions=index == 1 or index == 2)\n\n File \"/usr/local/lib/python3.12/concurrent/futures/thread.py\", line 59, in run\n result = self.fn(*self.args, **self.kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File \"/usr/local/lib/python3.12/concurrent/futures/thread.py\", line 93, in _worker\n work_item.run()\n\n File \"/usr/local/lib/python3.12/threading.py\", line 1012, in run\n self._target(*self._args, **self._kwargs)\n\n File \"/usr/local/lib/python3.12/threading.py\", line 1075, in _bootstrap_inner\n self.run()\n\n File \"/usr/local/lib/python3.12/threading.py\", line 1032, in _bootstrap\n self._bootstrap_inner()\n", encoded_attributes: None, cause: None, failure_info: Some(ApplicationFailureInfo(ApplicationFailureInfo { r#type: "_DeadlockError", non_retryable: false, details: None, next_retry_delay: None })) }), force_cause: Unspecified }
worker-1 | ERROR:temporalio.worker._workflow:Failed running eviction job, not evicting. Since eviction could not be processed, this worker cannot complete and the slot will remain forever used.
worker-1 | Traceback (most recent call last):
worker-1 | File "/app/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow.py", line 260, in _handle_activation
worker-1 | completion = await asyncio.wait_for(
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/usr/local/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
worker-1 | return await fut
worker-1 | ^^^^^^^^^
worker-1 | File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 59, in run
worker-1 | result = self.fn(*self.args, **self.kwargs)
worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1 | File "/app/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py", line 405, in activate
worker-1 | raise RuntimeError(
worker-1 | RuntimeError: Eviction processed, but 1 tasks remain. Stack traces below:
worker-1 |
worker-1 | Stack for <Task cancelling name='run (workflow: TaskExecutionWorkflow, id: 067c1696-664c-726a-8000-d6a82280fc51, run: d8fcd064-e206-480a-907a-42c08d41dec3)' coro=<_WorkflowInstanceImpl._run_top_level_workflow_function() running at /app/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py:1808> cb=[set.remove()]> (most recent call last):
worker-1 | File "/usr/local/lib/python3.12/threading.py", line 1032, in _bootstrap
worker-1 | self._bootstrap_inner()
worker-1 |
worker-1 | File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
worker-1 | self.run()
worker-1 |
worker-1 | File "/usr/local/lib/python3.12/threading.py", line 1012, in run
worker-1 | self._target(*self._args, **self._kwargs)
worker-1 |
worker-1 | File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 93, in _worker
worker-1 | work_item.run()
worker-1 |
worker-1 | File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 59, in run
worker-1 | result = self.fn(*self.args, **self.kwargs)
worker-1 |
worker-1 | File "/app/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py", line 377, in activate
worker-1 | self._run_once(check_conditions=index == 1 or index == 2)
worker-1 |
worker-1 | File "/app/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py", line 1785, in _run_once
worker-1 | handle._run()
worker-1 |
worker-1 | File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run
worker-1 | self._context.run(self._callback, *self._args)
worker-1 |
worker-1 | File "/app/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py", line 1808, in _run_top_level_workflow_function
worker-1 | await coro
worker-1 |
worker-1 | ||| [SERIALIZE] Time taken: 1.0680327415466309s // Object size: 0.072kb
👟 Reproduction steps
Run the following task:
tools:
- name: get_skin_collection_products
type: api_call
api_call:
method: GET
url: https://www.tirabeauty.com/ext/plpoffers/application/api/v1.0/collections/skin/items
main:
- tool: get_skin_collection_products
arguments:
params:
filters: false
page_id: 1
page_size: 1000
👀 Have you searched previous issues to see if this has been raised before?
- [x] I checked and didn't find similar issue
