Warning: `Activity not found on completion.`

[lorensr]

2023-01-15T22:11:14.904595Z WARN complete_activity: temporal_sdk_core::worker::activities: Activity not found on completion. This may happen if the activity has already been cancelled but completed anyway. task_token=TaskToken(CiRhZTE5Y2IwOC02NjViLTQyOTMtYmEyYi1lYmZkMjgyMWM5ZDcSSjUwYTMyNGE0LWQxMjItNDAwYS05YTNlLTEzMmU0OTc5ZWYxN19fOTJhZTc4ZGYtNzVkZi00NTBlLWFlODUtOTAyYjJiYzUyOWY4GiQzZGNhODQ4Zi1hMzUyLTQ5NWQtODUzYy1kMzkwYWFhMjkxMTUgEygBMgE1QgtzdGFydFNlYXJjaEoICAEQjJFAGAE=) details=Status { code: NotFound, message: "invalid activityID or activity already timed out or invoking workflow is completed", details: b"\x08\x05\x12Rinvalid activityID or activity already timed out or invoking workflow is completed\x1aB\[email protected]/temporal.api.errordetails.v1.NotFoundFailure", metadata: MetadataMap { headers: {"content-type": "application/grpc"} }, source: None } task_token=CiRhZTE5Y2IwOC02NjViLTQyOTMtYmEyYi1lYmZkMjgyMWM5ZDcSSjUwYTMyNGE0LWQxMjItNDAwYS05YTNlLTEzMmU0OTc5ZWYxN19fOTJhZTc4ZGYtNzVkZi00NTBlLWFlODUtOTAyYjJiYzUyOWY4GiQzZGNhODQ4Zi1hMzUyLTQ5NWQtODUzYy1kMzkwYWFhMjkxMTUgEygBMgE1QgtzdGFydFNlYXJjaEoICAEQjJFAGAE= status=ActivityExecutionResult(Failure(Failure(timeout of 1000ms exceeded, Application Failure: AxiosError))) task_queue=background-check workflow_id="50a324a4-d122-400a-9a3e-132e4979ef17__92ae78df-75df-450e-ae85-902b2bc529f8" run_id="3dca848f-a352-495d-853c-d390aaa29115"

I'm not sure why I'm getting this warning. Unlike in #445, I'm not canceling—I'm just running 3 activities in parallel. I don't think this applies either:

activity already timed out or invoking workflow is completed

It would also be nice if the warning included the Activity Id so I could match it against history.

Code:

https://github.com/temporalio/background-check-typescript/blob/911e7c47b7e37bf1dccfd4ee1317ef114b0cb765/src/workflows.ts#L20-L22

History:

https://www.dropbox.com/s/w0o3q2plz2bd2ad/3dca848f-a352-495d-853c-d390aaa29115_events.json?dl=0

Output:

Round ID: 51dea4dd-6871-43d0-9299-886757613875
2023-01-15T22:10:55.949176Z  INFO temporal_sdk_core::worker: Initializing worker task_queue=background-check namespace=default
Receiving HTTP request: POST /background with body: {
  customerId: '50a324a4-d122-400a-9a3e-132e4979ef17',
  userId: '92ae78df-75df-450e-ae85-902b2bc529f8',
  action: 'start'
}
📡 requestApproval response: { status: 'started' }
📡 getApprovalStatus response: { status: 'pending', confirmation: null }
📡 getApprovalStatus response: { status: 'pending', confirmation: null }
📡 getApprovalStatus response: { status: 'pending', confirmation: null }
📡 getApprovalStatus response: { status: 'pending', confirmation: null }
📡 getApprovalStatus response: {
  status: 'complete',
  confirmation: '4f1ebc3a-0c73-4f95-b8b1-2ea6a20e84d9'
}
📡 startSearch response: { status: 'started' }
📡 startSearch response: { status: 'started' }
📡 getSearchResult response: { status: 'pending', confirmation: null }
📡 getSearchResult response: { status: 'pending', confirmation: null }
2023-01-15T22:11:14.904595Z  WARN complete_activity: temporal_sdk_core::worker::activities: Activity not found on completion. This may happen if the activity has already been cancelled but completed anyway. task_token=TaskToken(CiRhZTE5Y2IwOC02NjViLTQyOTMtYmEyYi1lYmZkMjgyMWM5ZDcSSjUwYTMyNGE0LWQxMjItNDAwYS05YTNlLTEzMmU0OTc5ZWYxN19fOTJhZTc4ZGYtNzVkZi00NTBlLWFlODUtOTAyYjJiYzUyOWY4GiQzZGNhODQ4Zi1hMzUyLTQ5NWQtODUzYy1kMzkwYWFhMjkxMTUgEygBMgE1QgtzdGFydFNlYXJjaEoICAEQjJFAGAE=) details=Status { code: NotFound, message: "invalid activityID or activity already timed out or invoking workflow is completed", details: b"\x08\x05\x12Rinvalid activityID or activity already timed out or invoking workflow is completed\x1aB\[email protected]/temporal.api.errordetails.v1.NotFoundFailure", metadata: MetadataMap { headers: {"content-type": "application/grpc"} }, source: None } task_token=CiRhZTE5Y2IwOC02NjViLTQyOTMtYmEyYi1lYmZkMjgyMWM5ZDcSSjUwYTMyNGE0LWQxMjItNDAwYS05YTNlLTEzMmU0OTc5ZWYxN19fOTJhZTc4ZGYtNzVkZi00NTBlLWFlODUtOTAyYjJiYzUyOWY4GiQzZGNhODQ4Zi1hMzUyLTQ5NWQtODUzYy1kMzkwYWFhMjkxMTUgEygBMgE1QgtzdGFydFNlYXJjaEoICAEQjJFAGAE= status=ActivityExecutionResult(Failure(Failure(timeout of 1000ms exceeded, Application Failure: AxiosError))) task_queue=background-check workflow_id="50a324a4-d122-400a-9a3e-132e4979ef17__92ae78df-75df-450e-ae85-902b2bc529f8" run_id="3dca848f-a352-495d-853c-d390aaa29115"
📡 getSearchResult response: { status: 'pending', confirmation: null }
📡 getSearchResult response: { status: 'pending', confirmation: null }
📡 startSearch response: { status: 'started' }
📡 getSearchResult response: { status: 'pending', confirmation: null }
📡 getSearchResult response: { status: 'pending', confirmation: null }
📡 getSearchResult response: { status: 'pending', confirmation: null }
📡 getSearchResult response: { status: 'pending', confirmation: null }
📡 getSearchResult response: {
  status: 'complete',
  confirmation: '32b69451-9282-44e5-acc5-f90173c663f3'
}
📡 getSearchResult response: {
  status: 'complete',
  confirmation: 'aa5a4ac1-702f-44eb-923e-1708bbc2f25d'
}
📡 getSearchResult response: {
  status: 'complete',
  confirmation: '3f50a931-7bbc-4a68-8855-fd7e20f73113'
}