Workflow stuck when calling `workflow.GetVersion`

[seriousben]

Calling GetVersion results in a loop / weird behavior on the cadence server and the worker.

Code causing the behavior:

workflow.GetVersion(ctx, "step1", workflow.DefaultVersion, 1)

... rest of workflow

Cadence UI Screenshot:

Error in text:

Could not parse body (arg3) argument. Expected Thrift encoded arg3 for endpoint WorkflowService::GetWorkflowExecutionHistory. Got    instead of Thrift. Parsing error was: value must be a valid member of enumeration DecisionTaskFailedCause, got: 22.

Cadence server logs:

"msg":"invalid search attribute","service":"cadence-history","component":"shard-controller","shard-id":2,"component":"history-engine","es-mapping-key":"CadenceChangeVersion","wf-domain-name":"<redacted>","logging-call-at":"searchAttrValidator.go:79"
"msg":"Failing the decision.","service":"cadence-history","component":"shard-controller","shard-id":2,"component":"history-engine","wf-decision-fail-cause":22,"wf-id":"<redacted>","wf-run-id":"6f535980-dbd6-4731-8ea8-c9d2bd6566e5","wf-domain-id":"73e5c82e-d45f-4de6-95f4-25b5ce00f9bb","logging-call-at":"decisionHandler.go:447"

Versions:

Cadence server:

• Configured to use MySQL
• ubercadence/server:0.10.3-auto-setup

Cadence client:

• go.uber.org/cadence v0.10.5

---

The bug was initially reported on slack: https://uber-cadence.slack.com/archives/CM4K75Y7R/p1579622445006800

[martinxsliu]

I ran into a similar issue with adding GetVersion for the first time. Downgrading the client version from 0.10.5 to 0.9.3 worked (with server version 0.7.1)

[seriousben]

I was able to reproduce this bug with the samples: https://github.com/uber-common/cadence-samples/compare/master...seriousben:reproduce-get-version-bug?expand=1

In that branch, I've upgraded the go client, run go mod tidy and added the workflow.GetVersion line.

[seriousben]

Just ran more tests that seem to be pointing towards everything working if a new version of the server was released and if the cadence-web UI was also released.

Running the tweaked sample code (calling GetVersion) with the latest server docker image (master-auto-setup) works fine when testing with both the v0.10.5 and master version of the client.

But the latest release of the cadence-web docker image cannot show the history of a workflow that called GetVersion without erroring with:

Could not parse body (arg3) argument. Expected Thrift encoded arg3 for endpoint WorkflowService::GetWorkflowExecutionHistory. Got <some binary / weird chars> instead of Thrift. Parsing error was: value must be a valid member of enumeration EventType, got: 41. 

---

On a side note, github.com/apache/thrift keeps getting updated when doing a go get -u of cadence so I have to pin it to the version used in the client (and the server as well it looks like) which is go get -u github.com/apache/thrift@b2a4d4ae21c7 after changing the cadence version.

[vancexu]

Thanks @seriousben for reporting this.

1. We will release Cadence server 0.11 this week, and it contains a fix for the errors you mentioned.
2. Were you able to view history using CLI? If so it is a web issue only and @just-at-uber can you help this. (we may just need a release of Cadence Web)

[vancexu]

FYI we just released Cadence Web v3.6.0

[seriousben]

@vancexu I haven't tried with the CLI unfortunately. Do you know if the server will be released soon?

[vancexu]

@vancexu I haven't tried with the CLI unfortunately. Do know if the server will be released soon?

Sorry about postponing 0.11 release. We saw some degradation on activity timeout task and were working on it. Team needs more time to root cause and fix it.

[seriousben]

@vancexu Thanks a lot for the update.