Support for .NET CancellationTokens

[koenbeuk]

This PR allows for grain methods to be cancelled using a CancellationToken. The implementation is loosely based on the existing GrainCancellationToken support that is already available.

1. CodeGen detects the use of a CancellationToken in a grain interface method
2. If multiple CancellationTokens are used within the same grain interface method then ORLEANS0109 diagnostic is raised
3. The generated proxy throws early if cancellation is requested. Otherwise it will register for cancellation and sends a one-way message to ICancellableInvokableGrainExtension.CancelRemoteToken to mark this token as canceled
4. CancellableInvokableGrainExtension looks up the grains ICancellationRuntime component to mark the token as canceled
5. The generated invokable implements an additional interface: ICancellableInvokable which exposes the method: GetCancellableTokenId
6. Each instance of a CancellableInvokable sets a cancellableTokenId as Guid.NewGuid()
7. The InvokeInner implementation on the invokable can optionally interact with an ICancellationRuntime to track cancellation of this invokable.

Once the invokable completes, its CancellationTokenSource obtained through the ICancellationRuntime is always Canceled

Fixes: #8958

Microsoft Reviewers: Open in CodeFlow

[koenbeuk]

There is a regression with supporting IAsyncEnumerable with this PR. Will fix that regression and in turn, try and fix #8958

[kzu]

This would be super helpful 🙏

[ReubenBond]

@koenbeuk is there anything remaining here or is this ready for review? Apologies for the delay

[ReubenBond]

(rebased on main)

[ReubenBond]

@koenbeuk thank you for great work on this PR and for your patience. I have pushed some updates to implement the identification strategy we discussed (using sender's GrainId + MessageId).

I added two config options on MessagingOptions (i.e, SiloMessagingOptions/ClientMessagingOptions):

/// <summary>
/// Whether request cancellation should be attempted when a request times out.
/// </summary>
/// <remarks>
/// Request cancellation may involve sending a cancellation message to the silo which hosts the target grain.
/// Defaults to <see langword="true"/>.
/// </remarks>
public bool CancelRequestOnTimeout { get; set; } = true;

/// <summary>
/// Whether local calls should be cancelled immediately when cancellation is signaled (<see langword="false"/>)
/// rather than waiting for callees to acknowledge cancellation before a call is considered cancelled (<see langword="true"/>).
/// </summary>
/// <remarks>
/// Defaults to <see langword="false"/>.
/// </remarks>
public bool WaitForCancellationAcknowledgement { get; set; }

CancelRequestOnTimeout sends a cancellation signal in two cases:

• When the request times out and CallbackData.OnTimeout() is called.
• When a status message is received for an unknown request.

I'm not sure that we necessarily need to make these things configurable, but I erred on this side.

I changed the codegen to include a CancellationTokenSource object on the IInvokable implementation and to add a TryCancel() method to cancel it if the method accepts a CancellationToken. I excluded CancellationToken args from serialization.

I updated the IAsyncEnumerable impl to take advantage of this CancellationToken support.

I implemented batching for cancellation to hopefully reduce the impact of large volumes of cancellations on the wire - one might expect cancellation rates to increase when load increases. If that also leads to a significant increase in network traffic / CPU load, that could make the situation worse, so I felt that some batching was warranted.

CancellationToken parameters can be added to existing methods (as the last argument) without breaking forward or backwards compatibility.