uncaught exception in com.uber.cadence.internal.worker.PollerOptions

[tylerprete]

Getting this stack trace:

	at c.u.c.WorkflowService$RespondActivityTaskCompleted_result$RespondActivityTaskCompleted_resultStandardScheme.read(WorkflowService.java:26445)
	at c.u.c.WorkflowService$RespondActivityTaskCompleted_result$RespondActivityTaskCompleted_resultStandardScheme.read(WorkflowService.java:26413)
	at c.u.c.WorkflowService$RespondActivityTaskCompleted_result.read(WorkflowService.java:26315)
	at o.apache.thrift.TDeserializer.deserialize(TDeserializer.java:81)
	at o.apache.thrift.TDeserializer.deserialize(TDeserializer.java:67)
	at c.u.t.m.ThriftSerializer.decodeBody(ThriftSerializer.java:72)
	at c.u.t.messages.Serializer.decodeBody(Serializer.java:49)
	at c.u.t.messages.EncodedResponse.getBody(EncodedResponse.java:85)
	at c.u.c.s.WorkflowServiceTChannel.respondActivityTaskCompleted(WorkflowServiceTChannel.java:1170)
	at c.u.c.s.WorkflowServiceTChannel.lambda$RespondActivityTaskCompleted$15(WorkflowServiceTChannel.java:1157)
	at c.u.c.s.Work...

I have some Async activities running that waiting with Promise.allOf for.

There may be an underlying issue I need to fix in my activity, but it seems the java client isn't expecting this response from cadence the server and is handling it poorly.

[tylerprete]

Also sometimes seeing

c.u.c.EntityNotExistsError: Decision task not found.\n\tat c.u.c.WorkflowService$RespondDecisionTaskCompleted_result$RespondDecisionTaskCompleted_resultStandardScheme.read(WorkflowService.java:19244)\n\tat c.u.c.WorkflowService$RespondDecisionTaskCompleted_result$RespondDecisionTaskCompleted_resultStandardScheme.read(WorkflowService.java:19203)\n\tat c.u.c.WorkflowService$RespondDecisionTaskCompleted_result.read(WorkflowService.java:19094)\n\tat o.apache.thrift.TDeserializer.deserialize(TDeserializer.java:81)\n\tat o.apache.thrift.TDeserializer.deserialize(TDeserializer.java:67)\n\tat c.u.t.m.ThriftSerializer.decodeBody(ThriftSerializer.java:72)\n\tat c.u.t.messages.Serializer.decodeBody(Serializer.java:49)\n\tat c.u.t.messages.EncodedResponse.getBody(EncodedResponse.java:85)\n\tat c.u.c.s.WorkflowServiceTChannel.respondDecisionTaskCompleted(WorkflowServiceTChannel.java:928)\n\tat c.u.c.s.WorkflowServiceTChannel.lambda$RespondDecisionTaskCompleted$10(WorkflowServiceTChannel.java:915)\n\tat c.u.c.s.Workfl...

This is on a cron job that uses Async tasks internally, for some extra context.

Let me know if there is additional information I can provide.

[mfateev]

The first stack trace appears incomplete. Could you post the complete chain? The second happens when a decision task takes longer than the decision task timeout. Do you do any expensive operations in the workflow body? What is the decision task timeout?

[tylerprete]

Unfortunately I didn't abbreviate that -- by the time it ended up in the logs, it was abbreviated. I may be able to change our logback.xml but that would require a deployment, etc.

I don't think I'm doing anything expensive in the workflow -- I just spin off a bunch of async activities (up to ~150), then wait on them all.

Decision task timeout is still default. I'm only specifying scheduleToCloseTimeout for the workflow and activities.