ContinueAsNew should create new root span

[bincyber]

Is your feature request related to a problem? Please describe.

Currently, when tracing is enabled on a workflow that ends as ContinueAsNew, the new workflow is attached to the existing trace:

The total number of spans can become extremely large and overwhelm the tracing backend (eg, Grafana Tempo, Jaeger, etc.).

Describe the solution you'd like

It would be better if the new workflow was created as a new, disconnected trace and linked to the existing trace:

This idea can be extended to disconnecting Child Workflows as well.

Additional context

Slack: https://temporalio.slack.com/archives/CTDTU3J4T/p1702921125553459

[Quinn-With-Two-Ns]

Added a note to discuss this internally across the other SDKs to see what/if they do to support this

[Quinn-With-Two-Ns]

Looking at some other SDKs they don't provide any convenient option for this. There are a lot of reasons we could want to create a disconnected span , like continue as new, or as you said a child workflow. Some other reasons brought up are Cron and schedules. A user may also only want to disconnect them if the size if growing particularly large or for certain workflow types. These are to many options to support simple booleans for.

For the most flexibility I wonder if we should just expose the ability to get the otel span from the workflow context, then users can interact with the span like they do outside of workflow code.

[Quinn-With-Two-Ns]

Users can then develop their own interceptors for common cases like disconnect spans on continue as new

[ohad83]

I encountered a similar error (in my case for child workflows). If using the otel tracing interceptor you could pass a customSpanStarter to do that. This is the one I created:

	if strings.HasPrefix(spanName, "RunWorkflow:") && trace.SpanFromContext(ctx).SpanContext().IsValid() {
		opts = append(opts,
			trace.WithNewRoot(), trace.WithLinks(trace.LinkFromContext(ctx)),
		)
	}
	_, span := t.Start(ctx, spanName, opts...)
	return span

It's not perfect since the link is unidirectional, so the child is linked to the parent but the parent isn't linked to the child (afaik it can't be, since the ExecuteChildWorkflow span finishes before the RunWorkflow one starts). It also means you won't see the duration of the child workflow in the parent trace, since the ExecuteChildWorkflow is short.

It might be possible to wrap each RunWorkflow in 2 spans, one in the parent trace and one as the root span of a child trace, and link them, and also only finish the one in the parent when the child finishes. It requires a bit more tinkering but I think it's possible without fully implementing a new interceptor.