azure-functions-host icon indicating copy to clipboard operation
azure-functions-host copied to clipboard

Published 20 hours ago • verified publisher icon Azure

COMException and RpcException in Function App

Open HShather opened this issue 5 years ago • 16 comments

We have had a couple of incidents recently where our function apps start erroring badly. We see a high percentage of failures (many 1000's) across all functions in the app caused by System.Runtime.InteropServices.COMException and Microsoft.Azure.WebJobs.Script.Rpc.RpcException. The only way we have found to fix the issue is to restart the function app after which point the errors completely stop. Interestingly, we have seen that this incident seems to affect all function apps running on a given app service plan. The errors seem to be coming from the function runtime. It would be useful to know if this is caused by instability in the func runtime or if it is something we are doing wrong.

Our function apps are written in Node, run the latest V2 runtime and run on app service plans. Most of our functions are http triggers.

Investigative information

The last incident started on 2019-03-10 2100 UTC and lasted around 14 hours (at which point we restarted the app and the errors stopped)

Com Exception Example

  • Timestamp: 2019-03-11T09:30:59.8876066Z
  • Function App version: 2.0
  • Invocation ID: 515cf6a6-79c9-46dc-bb62-db6fb1d02be1
  • Region: SEA

Rpc Exception Example

  • Timestamp: 2019-03-11T07:50:35.8048046Z
  • Function App version: 2.0
  • Invocation ID: 148bd6f4-2fda-4658-a446-48eca495451e
  • Region: SEA

HShather avatar Mar 13 '19 08:03 HShather

@HShather - I looked into those invocation ID's, and the inner details seem to point to unexpected behavior in the function code.

The RpcException is caused by a Not Authenticated MongoError. For more details on where on in code you can check for errors, the RpcException contains a stack trace in "Details" (you should be able to this in App Insights, let me know if not!).

The details on the COMException say "an operation was attempted on a nonexistent network connection." From a platform point of view, your connections look healthy. Could the client making a call to the function be disconnecting mid-request?

mhoeger avatar Mar 19 '19 18:03 mhoeger

Thanks for looking at this @mhoeger

With regard to the COMException This is a mobile api but all calls are routed through application gateway and then API Management before being served by the function app.

Could the client making a call to the function be disconnecting mid-request?

We saw many 1000's of this error so I dont think it could be mobile clients. Could it be something to do with APIM integration with function apps. Also, how do you explain the fact that a restart of the function app stopped the error?

With regard to the RPCException

The RpcException is caused by a Not Authenticated MongoError. For more details on where on in code you can check for errors, the RpcException contains a stack trace in "Details"

These functions do talk with a cosmos db over mongo api but I dont see any MongoError in our appinsights. I've attached the stack trace the we see in app insights for invocation id 148bd6f4-2fda-4658-a446-48eca495451e. Is there somewhere else where this will be logged? RPCException StackTrace.txt

Another thing of interest that I mentioned in the original post was the issue simultaneously affected all function apps running on the same app service plan. Below are the details of another RPCException that affected a second function app running on the same app service plan. If you can see the telemetry of the whole incident you will see that the errors on the 2nd function app started at exactly the same time as the first function app and then stopped with an app restart. Again, if we were looking at an issue with cosmos db, then why would a restart of the function app fix the issue? I can't help but think both exception types are related and have the same root cause.

  • Timestamp: 2019-03-11T00:48:34.7872327Z
  • Function App version: 2.0
  • Invocation ID: 328e7d4b-ad1c-44ff-958b-a999d0c73aed
  • Region: SEA

HShather avatar Mar 20 '19 10:03 HShather

@HShather - I'll look into why the RpcException isn't showing fully in App Insights, but for now, here are the details:

RpcException details Microsoft.Azure.WebJobs.Script.Rpc.RpcException : Result: Failure Exception: MongoError: Not Authenticated Stack: MongoError: Not Authenticated at D:\home\site\wwwroot\index.js:358479:63 at authenticateStragglers (D:\home\site\wwwroot\index.js:358402:16) at Connection.messageHandler (D:\home\site\wwwroot\index.js:358438:5) at emitMessageHandler (D:\home\site\wwwroot\index.js:97443:10) at TLSSocket. (D:\home\site\wwwroot\index.js:97586:17) at TLSSocket.emit (events.js:182:13) at TLSSocket.EventEmitter.emit (domain.js:442:20) at addChunk (_stream_readable.js:283:12) at readableAddChunk (_stream_readable.js:264:11) at TLSSocket.Readable.push (_stream_readable.js:219:10) at TLSWrap.onread (net.js:638:20) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Description\Rpc\WorkerLanguageInvoker.cs : 74 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Description\FunctionInvokerBase.cs : 84 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task`1 src) at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Description\FunctionGenerator.cs : 225 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs : 63 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 556 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstance instance,ParameterHelper parameterHelper,ILogger logger,CancellationTokenSource functionCancellationTokenSource) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 503 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 439 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 249

All of the RpcExceptions are coming from a single host instance, meaning a single running function app instance (host instance id = 2d81f304-30bc-47fe-930c-30baceec7314). You're right that all functions in the same role instance were affected, but the important commonality is that all of your code that hit this RpcException was in the same state (running on the same function app instance). My guess is that something happened in that code (perhaps a race condition) that made the mongo client become unauthenticated.

I will continue looking at the COMException, although it does seem likely that you may need to reach out to the API Management team since they are the client in the eyes of the functions host.

mhoeger avatar Mar 20 '19 23:03 mhoeger

@HShather - any updates on this issue?

mhoeger avatar Mar 29 '19 23:03 mhoeger

No progress yet unfortunately, we still see sporadic incidences of this type

The RPCException/Mongo exception - we dont see the mongo exception in our telemetry, just the RPC one. It makes we wonder what else we are missing. Is this an issue with functions app insights integration?I have asked the dev team to look again at the error handling in the codebase Moreover, our stack traces are not helpful e.g. Not Authenticated at D:\home\site\wwwroot\index.js:358479:63 - is not very useful the code is minified using webpack. According to the development team, there is no support for source maps in functions which prevents them linking this back to the original code.

ComException - I raised this feedback with APIM team https://feedback.azure.com/forums/248703-api-management/suggestions/37252213-comexception-when-integrating-with-azure-functions

HShather avatar Apr 01 '19 09:04 HShather

The RPCException here is expected (that will be the exception type you'll see wrapping the invocation error), the root cause is the Mongo error being thrown.

@mhoeger any additional follow up needed on the App Insights investigation? @brettsam has some thoughts here.

fabiocav avatar Apr 10 '19 18:04 fabiocav

@mhoeger Any updates to this issue? We have set up some functions and seeing the exact same issue. On a load test, after about 300 invocations (each of which makes a call to Mongo), all functions completely crash until restarted. After another 300 or so requests, same issue. We are on a Premium App Service plan.

Here are the details of the exception we are seeing: Microsoft.Azure.WebJobs.Host.FunctionInvocationException: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__19.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.9.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.9.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.9.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 321) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsyncCore>d__16.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.9.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.9.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.9.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 117) Inner exception Microsoft.Azure.WebJobs.Script.Rpc.RpcException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw`

And the reason being: Exception while executing function: Functions.sync-service Result: Failure Exception: MongoNetworkError: connect ENOBUFS *removed* - Local (undefined:undefined) Stack: Error: connect ENOBUFS *removed* - Local (undefined:undefined) at internalConnect (net.js:884:16) at defaultTriggerAsyncIdScope (internal/async_hooks.js:294:19) at defaultTriggerAsyncIdScope (net.js:974:9) at process._tickCallback (internal/process/next_tick.js:61:11)

I am also suspecting this isn't a Mongo issue because restarting the function fixes it.

utsavized avatar Jun 07 '19 15:06 utsavized

I'm having a similar issue to this but very different circumstances. I have a blob trigger running on an Node 10 Functions 2 app, that does some analysis and then makes a call to an Elasticsearch cluster to index a document related to the blob. It also was doing some other HTTP requests to retrieve a data file from blob storage separately (using the @azure/blob-storage package).

In my limited research, I suspect the ENOBUFS error is happening due to all the outgoing HTTP requests but I am unsure how to solve that as the HTTP requests are all being created/managed through third party code.

ocdi avatar Sep 17 '20 03:09 ocdi

I have the similar issuse

Exception type Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException

pabassam avatar Jan 21 '22 10:01 pabassam

Found this one too in my PowerShell Function App, it's scheduled but it gave this error only yesterday (once), it has a failure rate of 0.7%, not sure what the issue is.

Full Exception :
 Exception while executing function: removedName ---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException : Result: Failure
Exception: The running command stopped because the preference variable "ErrorActionPreference" or common parameter is set to Stop: 5:51:52 AM - Error: Code=; Message=A task was canceled.

Stack:    at System.Management.Automation.Runspaces.PipelineBase.Invoke(IEnumerable input)
   at System.Management.Automation.PowerShell.Worker.ConstructPipelineAndDoWork(Runspace rs, Boolean performSyncInvoke)
   at System.Management.Automation.PowerShell.Worker.CreateRunspaceIfNeededAndDoWork(Runspace rsToUse, Boolean isSync)
   at System.Management.Automation.PowerShell.CoreInvokeHelper[TInput,TOutput](PSDataCollection`1 input, PSDataCollection`1 output, PSInvocationSettings settings)
   at System.Management.Automation.PowerShell.CoreInvoke[TInput,TOutput](PSDataCollection`1 input, PSDataCollection`1 output, PSInvocationSettings settings)
   at System.Management.Automation.PowerShell.CoreInvoke[TOutput](IEnumerable input, PSDataCollection`1 output, PSInvocationSettings settings)
   at System.Management.Automation.PowerShell.Invoke[T](IEnumerable input, IList`1 output, PSInvocationSettings settings)
   at System.Management.Automation.PowerShell.Invoke[T]()
   at Microsoft.Azure.Functions.PowerShellWorker.PowerShell.PowerShellExtensions.InvokeAndClearCommands[T](PowerShell pwsh) in /mnt/vss/_work/1/s/src/PowerShell/PowerShellExtensions.cs:line 53
   at Microsoft.Azure.Functions.PowerShellWorker.PowerShell.PowerShellManager.InvokeNonOrchestrationFunction(DurableController durableController, IDictionary outputBindings) in /mnt/vss/_work/1/s/src/PowerShell/PowerShellManager.cs:line 304
   at Microsoft.Azure.Functions.PowerShellWorker.PowerShell.PowerShellManager.InvokeFunction(AzFunctionInfo functionInfo, Hashtable triggerMetadata, TraceContext traceContext, RetryContext retryContext, IList`1 inputData, FunctionInvocationPerformanceStopwatch stopwatch) in /mnt/vss/_work/1/s/src/PowerShell/PowerShellManager.cs:line 255
   at Microsoft.Azure.Functions.PowerShellWorker.RequestProcessor.InvokeFunction(AzFunctionInfo functionInfo, PowerShellManager psManager, FunctionInvocationPerformanceStopwatch stopwatch, InvocationRequest invocationRequest) in /mnt/vss/_work/1/s/src/RequestProcessor.cs:line 334
   at Microsoft.Azure.Functions.PowerShellWorker.RequestProcessor.ProcessInvocationRequestImpl(StreamingMessage request, AzFunctionInfo functionInfo, PowerShellManager psManager, FunctionInvocationPerformanceStopwatch stopwatch) in /mnt/vss/_work/1/s/src/RequestProcessor.cs:line 323
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at /_/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs : 96
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at /_/src/WebJobs.Script/Description/FunctionInvokerBase.cs : 82
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2.InvokeAsync[TReflected,TReturnType](TReflected instance,Object[] arguments) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\VoidTaskMethodInvoker.cs : 20
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs : 52
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 581
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance,ParameterHelper parameterHelper,ILogger logger,CancellationTokenSource functionCancellationTokenSource) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 527
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 306
   End of inner exception
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 352
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 108

erwinkramer avatar Jul 22 '22 10:07 erwinkramer

Hi, I am struggling with the same issue. Not sure how to make it work again. Did anyone find any solutions on this error? Even when I run the python code locally, I get error: ValueError: Missing input parameter client_id Looking forward to hearing for solutions.

Microsoft.Azure.WebJobs.Host.FunctionInvocationException: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.34.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:352) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsync>d__18.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.34.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:108) Inner exception Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker+<InvokeCore>d__9.MoveNext (Microsoft.Azure.WebJobs.Script, Version=3.11.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:96) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+<Invoke>d__24.MoveNext (Microsoft.Azure.WebJobs.Script, Version=3.11.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script/Description/FunctionInvokerBase.cs:82) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator+<Coerce>d__31.MoveNext (Microsoft.Azure.WebJobs.Script, Version=3.11.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script/Description/FunctionGenerator.cs:225) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker2+<InvokeAsync>d__10.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.34.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:52) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeWithTimeoutAsync>d__33.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.34.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:581) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__32.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.34.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:527) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.34.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:306)

atharvajdhumal avatar Aug 16 '22 12:08 atharvajdhumal

Any updates on this? I too have this issue.

Lwhieldon avatar Jan 12 '23 20:01 Lwhieldon

Still no updates on this?

julienFlexsoft avatar Sep 05 '23 15:09 julienFlexsoft

Any updates on this, facing similar issue!! Any help could make it better for me.

ankit-gautam23 avatar Jan 16 '24 13:01 ankit-gautam23

Any updates on this? I am getting a Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException error only for a specific timeframe (10pm-12pm), it works the rest of the day.

rapha-etos avatar Feb 27 '24 11:02 rapha-etos