Nil pointer exception during local activity retry

[yycptt]

When retrying local activity, a new goroutine will be created through time.AfterFunc. In this goroutine, workflowExecutionContext.eventHandler will be accessed first, and then the local activity will be sent to a channel for processing.

However, if during the retry backoff duration decision heartbeat failed, the state workflowExecutionContext will be cleared and the eventHandler will be nil. This results in a nil pointer exception when the new goroutine got executed.

The fix is basically check if eventHandler is nil or not before accessing it. However, since it's in another goroutine, we need to grab a lock. However, during the entire decision task processing, lock on workflowExecutionContext is locked so we can't grab the lock again. So in the temporally fix https://github.com/uber-go/cadence-client/pull/913 the type of eventHandler is changed to atomic.Value so it can be accessed in multiple goroutines without race condition.

Ideally, the retry timer should be part of the event loop for processing the workflow task and a component for managing all the retry timers is needed.