Support distributed tracing for Orleans Reminders

[oleksandr-bilyk]

Would be cool if Orleans reminders will support System.Diagnostics.Activity.Current.Id is used to track the activity in distributed tracing.

It will flow from registration time and then to execution time. I did it manually with DAPR Actor reminders. It must be just one more string correlation id implicitly parameter in scheduler database.

Probably makes sense, initially, to make this feature disabled by default to avoid breaking changes but let Orleans Reminder config to enable it explicitly.

I may show sample. Implementation may be simple take current activity Id and put it into state store

SystemDiagnosticsActivityId =
    match System.Diagnostics.Activity.Current with
    | null -> "" // special case when activity is turned off
    | activity -> activity.Id |> defaultIfNull "" // special case when activity id is null

During reminder execution activity must be

let activityImpersonated (systemDiagnosticsActivityId: string) (action: unit -> Task<'TResult>) = task {
    use activityOriginal =
        // external thread acvivity will record that it is going to start new activity from another distributed tracing context.
        activitySource
            .StartActivity("StartingImpersonatedActivity")
            .AddTag("WorkflowActorMessageActivityImpersonated", systemDiagnosticsActivityId)
    Debug.Assert(Activity.Current = activityOriginal, "Creation of activity will make it current.")
    use activityImpersonated =
        activitySource
            .CreateActivity("ActivityImpersonation", ActivityKind.Internal)
            .SetParentId(systemDiagnosticsActivityId)
            .Start()
    Debug.Assert(Activity.Current = activityImpersonated, "Impersonated activity is current now.")
    try
        let! result = action()
        systemDiagnosticsActivityStatusOk(activityImpersonated)
        return result
    with exc ->
        systemDiagnosticsActivityStatusException(activityImpersonated, exc)
        return exc |> AggregateException |> raise
}

@ReubenBond