aws-lambda-dotnet
aws-lambda-dotnet copied to clipboard
Use ILambdaContext.RemainingTime to create a valid CancellationToken for AspNetCoreServer Requests
Describe the feature
Currently when a Lambda execution times out, a AspNetCore request continues to execute even if it is checking a CancellationToken parameter for cancellation.
This feature would create a new CancellationTokenSource using ILambdaContext.RemainingTime and request cancellation when remaining time reaches or nears zero. This would allow consumer code to cancel long running operations in a way that is more controlled. This would create similar functionality to how timeouts are handled in Kestrel and IIS
Use Case
Due to upstream concurrency limitations, we sometimes have a lambda AspNetCore request timeout. Since the CancellationToken passed into the request is never cancelled, we continue to try to execute retry requests in the background if a lambda is reused.
Proposed Solution
No response
Other Information
No response
Acknowledgements
- [X] I may be able to implement this feature request
- [ ] This feature might incur a breaking change
AWS .NET SDK and/or Package version used
Amazon.Lambda.AspNetCoreServer.Hosting 1.5.0
Targeted .NET Platform
net6.0
Operating System and version
AmazonLinux (Lambda)
Needs review with the team.
Sample implementation:
builder.Services.Configure<MvcOptions>(options =>
{
var existingBinders = options.ModelBinderProviders
.Where(x => x.GetType() == typeof(CancellationTokenModelBinderProvider)).ToArray();
foreach (var binder in existingBinders)
{
options.ModelBinderProviders.Remove(binder);
}
options.ModelBinderProviders.Insert(0, new LambdaRemainingTimeCancellationTokenModelBinderProvider());
});
public class LambdaRemainingTimeCancellationTokenModelBinderProvider : IModelBinderProvider
{
public IModelBinder? GetBinder(ModelBinderProviderContext context)
{
ArgumentNullException.ThrowIfNull(context);
if (context.Metadata.ModelType == typeof(CancellationToken))
{
return new LambdaRemainingTimeCancellationTokenModelBinder();
}
return null;
}
}
public class LambdaRemainingTimeCancellationTokenModelBinder : IModelBinder
{
private const string HttpContextItemKey = "LambdaContextRemainingTimeCancellationTokenSource";
private static readonly CancellationTokenModelBinder fallbackBinder = new();
public Task BindModelAsync(ModelBindingContext bindingContext)
{
ArgumentNullException.ThrowIfNull(bindingContext);
if (bindingContext.HttpContext.Items.ContainsKey(AbstractAspNetCoreFunction.LAMBDA_CONTEXT))
{
if (bindingContext.HttpContext.Items[AbstractAspNetCoreFunction.LAMBDA_CONTEXT] is ILambdaContext lambdaContext)
{
var cancellationTokenSource = new CancellationTokenSource(lambdaContext.RemainingTime);
// Add the CTS to the Request Context, so GC doesn't clean it up
bindingContext.HttpContext.Items.Add(HttpContextItemKey, cancellationTokenSource);
bindingContext.ValidationState.Add(cancellationTokenSource.Token,
new ValidationStateEntry() { SuppressValidation = true });
bindingContext.Result = ModelBindingResult.Success(cancellationTokenSource.Token);
return Task.CompletedTask;
}
}
// The Lambda Context is missing. We must not be running in Lambda, so fallback to the default behavior.
return fallbackBinder.BindModelAsync(bindingContext);
}
}
Here is my implementation of this. 1 second buffer to shutdown gracefully. Defaults to 15 minutes when LambdaContext?.RemainingTime is null
WARNING - Might have a bug with GC, see message from brianfeucht
public static class ExtensionMethods
{
public static CancellationTokenSource CreateCancellationTokenSource(this HttpContext instance, TimeSpan? timeoutBuffer = null)
{
var lambdaContext = instance.GetLambdaContext();
return lambdaContext.CreateCancellationTokenSource(timeoutBuffer ?? TimeSpan.FromSeconds(1));
}
private static CancellationTokenSource CreateCancellationTokenSource(this ILambdaContext instance, TimeSpan buffer)
{
if (instance.RemainingTime == TimeSpan.MaxValue)
throw new ArgumentOutOfRangeException(nameof(instance.RemainingTime));
if (buffer.TotalSeconds < 0)
throw new ArgumentOutOfRangeException(nameof(buffer));
var remainingTime = instance.RemainingTime.Subtract(buffer);
return remainingTime.TotalSeconds > 0
? new CancellationTokenSource(remainingTime)
: throw new InvalidOperationException(@"Insufficient Time Remaining");
}
private static ILambdaContext GetLambdaContext(this HttpContext instance)
{
const string key = @"LambdaContext";
if (!AWSXRayRecorder.IsLambda())
return new LocalLambdaContext(TimeSpan.FromMinutes(15));
instance.Items.TryGetValue(key, out var lambdaContext);
return lambdaContext as ILambdaContext
?? throw new InvalidOperationException($@"{nameof(HttpContext)}.{nameof(HttpContext.Items)}[{key}] is not {nameof(ILambdaContext)}");
}
}