DurableFunctionsMonitor icon indicating copy to clipboard operation
DurableFunctionsMonitor copied to clipboard
verified publisher icon microsoft

DfMon for MSSQL storage provider as NuGet package

Open mmajcica opened this issue 11 months ago • 6 comments

Hi all,

would it be possible to package the version for SQL as nuget package and publish it? Or maybe include the necessary in the main package and be able to activate a custom provider via a setting?

Thanks

mmajcica avatar Jan 22 '25 06:01 mmajcica

The MSSQL storage provider is already supported by the Durable Functions Monitor project. More information here: https://github.com/microsoft/DurableFunctionsMonitor/tree/main/custom-backends/mssql.

cgillum avatar Jan 29 '25 00:01 cgillum

The MSSQL storage provider is already supported by the Durable Functions Monitor project. More information here: https://github.com/microsoft/DurableFunctionsMonitor/tree/main/custom-backends/mssql.

Indeed I do know it is available. But it is not deployable via a nuget package as standard dfm. It would be nice if this backend was also packaged and published to make deployment easier. In certain enterprises, other deployment methods (suggested in the documentation) are not viable.

mmajcica avatar Jan 30 '25 07:01 mmajcica

@mmajcica , please, clarify whether you need the isolated version or the inproc version to be packed. Or both?

My guess is, since you only want to deploy that package (as a standalone instance), then isolated would be good enough, but please confirm.

scale-tone avatar Feb 03 '25 16:02 scale-tone

@mmajcica , please, clarify whether you need the isolated version or the inproc version to be packed. Or both?

My guess is, since you only want to deploy that package (as a standalone instance), then isolated would be good enough, but please confirm.

It does not mater for me, whatever it is easier to you. The only thing is that the currently available/suggested deployment methods do not work for me as there are certain company wide policies that are preventing me to deploy it. As we are using the DFM nuget package and that is working like a charm, it would be nice to have also this custom version with sql support so that we can finally move our durable functions to sql provider.

Thank you

mmajcica avatar Feb 05 '25 12:02 mmajcica

@mmajcica , I just pushed DurableFunctionsMonitor.DotNetIsolated.MsSql/6.6.0-beta4, can I ask you to check if it works for your scenario?

It expects SQL connection string as DFM_SQL_CONNECTION_STRING setting. You can use the same ARM template to deploy it.

scale-tone avatar Feb 07 '25 00:02 scale-tone

@mmajcica , I just pushed DurableFunctionsMonitor.DotNetIsolated.MsSql/6.6.0-beta4, can I ask you to check if it works for your scenario?

It expects SQL connection string as DFM_SQL_CONNECTION_STRING setting. You can use the same ARM template to deploy it.

I will test it this week, thanks a ton for such quick solution.

mmajcica avatar Feb 10 '25 07:02 mmajcica

Added in v6.7

scale-tone avatar May 20 '25 22:05 scale-tone

I just set up the 6.7.0 and once it loads I do get the following error: Load failed. One or more errors occurred. (Status(StatusCode="ResourceExhausted", Detail="Received message exceeds the maximum configured message size."))

Image

Any idea what that can be about?

mmajcica avatar May 21 '25 14:05 mmajcica

I have tried both with 6.6.0 beta for and the newest 6.7.0 These are my settings:

Image

mmajcica avatar May 21 '25 14:05 mmajcica

@mmajcica are you running this against a SQL database that has existing data? If so, is it possible that some existing data might be too large to be loaded by DfMon? This exception message is typically seen when attempting to load a payload that exceeds the default gRPC message limit of 4 MB.

cgillum avatar May 21 '25 22:05 cgillum

It is a brand new db, used only for df.

On Thu, 22 May 2025, 00:21 Chris Gillum, @.***> wrote:

cgillum left a comment (microsoft/DurableFunctionsMonitor#232) https://github.com/microsoft/DurableFunctionsMonitor/issues/232#issuecomment-2899392839

@mmajcica https://github.com/mmajcica are you running this against a SQL database that has existing data? If so, is it possible that some existing data might be too large to be loaded by DfMon? This exception message is typically seen when attempting to load a payload that exceeds the default gRPC message limit of 4 MB.

— Reply to this email directly, view it on GitHub https://github.com/microsoft/DurableFunctionsMonitor/issues/232#issuecomment-2899392839, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXC454LZPR2UKXIVULOBZT27T37VAVCNFSM6AAAAABVUEYW2WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQOJZGM4TEOBTHE . You are receiving this because you were mentioned.Message ID: @.***>

mmajcica avatar May 22 '25 04:05 mmajcica

I found a stack trace in the logs, maybe it helps. I'm not sure how can a standard use of DF create a message that exceeds that limit.

I'm trying to dig into this issue, this is how the size of my DB looks like

Image

Not sure how to get a single row size to see if I effectively have a row that is larger than 4MB (which I doubt).

Result: DFM failed Exception: System.AggregateException: One or more errors occurred. (Status(StatusCode="ResourceExhausted", Detail="Received message exceeds the maximum configured message size.")) ---> Grpc.Core.RpcException: Status(StatusCode="ResourceExhausted", Detail="Received message exceeds the maximum configured message size.") at Grpc.Net.Client.Internal.StreamExtensions.ReadMessageAsync[TResponse](Stream responseStream, GrpcCall call, Func2 deserializer, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken) at Grpc.Net.Client.Internal.GrpcCall2.ReadMessageAsync(Stream responseStream, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken) at Grpc.Net.Client.Internal.GrpcCall2.RunCall(HttpRequestMessage request, Nullable1 timeout) at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.<>c__DisplayClass18_0.<<GetAllInstancesAsync>b__0>d.MoveNext() --- End of stack trace from previous location --- at Microsoft.DurableTask.Pageable.FuncAsyncPageable1.AsPagesCore(String continuationToken, Nullable1 pageSizeHint, CancellationToken cancellation)+MoveNext() at Microsoft.DurableTask.Pageable.FuncAsyncPageable1.AsPagesCore(String continuationToken, Nullable1 pageSizeHint, CancellationToken cancellation)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult() at Microsoft.DurableTask.AsyncPageable1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext() at Microsoft.DurableTask.AsyncPageable1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext() at Microsoft.DurableTask.AsyncPageable1.GetAsyncEnumerator(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult() at System.Threading.Tasks.ValueTask1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state) --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at DurableFunctionsMonitor.DotNetIsolated.ExtensionMethodsForOrchestrations.ExpandStatus(IAsyncEnumerable1 asyncEnumerable, DurableTaskClient client, String connName, String hubName, FilterClause filterClause, HashSet1 hiddenColumns, DfmExtensionPoints extensionPoints)+MoveNext() in /home/runner/work/DurableFunctionsMonitor/DurableFunctionsMonitor/durablefunctionsmonitor.dotnetisolated.core/Functions/Orchestrations.cs:line 74 at DurableFunctionsMonitor.DotNetIsolated.ExtensionMethodsForOrchestrations.ListDurableEntities(IEnumerable1 orchestrations, DurableTaskClient client, Nullable1 timeFrom, Nullable1 timeTill, String[] statuses, HashSet1 hiddenColumns, ILogger log)+MoveNext() in /home/runner/work/DurableFunctionsMonitor/DurableFunctionsMonitor/durablefunctionsmonitor.dotnetisolated.core/Functions/Orchestrations.cs:line 203 at DurableFunctionsMonitor.DotNetIsolated.FilterClauseExtensions.ApplyFilter[T](IEnumerable1 items, FilterClause filter)+MoveNext() in /home/runner/work/DurableFunctionsMonitor/DurableFunctionsMonitor/durablefunctionsmonitor.dotnetisolated.core/Common/FilterClause.cs:line 197 at System.Collections.Generic.EnumerableHelpers.ToArray[T](IEnumerable1 source, Int32& length) at System.Linq.Buffer1..ctor(IEnumerable1 source) at System.Linq.OrderedEnumerable1.GetEnumerator(Int32 minIdx, Int32 maxIdx)+MoveNext() at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty) at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty) at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType) at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType) at Newtonsoft.Json.JsonSerializer.Serialize(JsonWriter jsonWriter, Object value, Type objectType) at Newtonsoft.Json.JsonConvert.SerializeObjectInternal(Object value, Type type, JsonSerializer jsonSerializer) at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, Type type, JsonSerializerSettings settings) at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, JsonSerializerSettings settings) at DurableFunctionsMonitor.DotNetIsolated.Globals.ReturnJson(HttpRequestData req, Object result, Func2 applyThisToJson) in /home/runner/work/DurableFunctionsMonitor/DurableFunctionsMonitor/durablefunctionsmonitor.dotnetisolated.core/Common/Globals.cs:line 127 at Dfm.DotNetIsolatedMsSql.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in /home/runner/work/DurableFunctionsMonitor/DurableFunctionsMonitor/custom-backends/dotnetIsolated-mssql/obj/Release/net8.0/Microsoft.Azure.Functions.Worker.Sdk.Generators/Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator/GeneratedFunctionExecutor.g.cs:line 128 at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13 at DurableFunctionsMonitor.DotNetIsolated.ExtensionMethods.<>c__DisplayClass0_0.<<UseDurableFunctionsMonitor>b__1>d.MoveNext() in /home/runner/work/DurableFunctionsMonitor/DurableFunctionsMonitor/durablefunctionsmonitor.dotnetisolated.core/Common/ExtensionMethods.cs:line 107 Stack: at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at DurableFunctionsMonitor.DotNetIsolated.ExtensionMethodsForOrchestrations.ExpandStatus(IAsyncEnumerable1 asyncEnumerable, DurableTaskClient client, String connName, String hubName, FilterClause filterClause, HashSet1 hiddenColumns, DfmExtensionPoints extensionPoints)+MoveNext() in /home/runner/work/DurableFunctionsMonitor/DurableFunctionsMonitor/durablefunctionsmonitor.dotnetisolated.core/Functions/Orchestrations.cs:line 74 at DurableFunctionsMonitor.DotNetIsolated.ExtensionMethodsForOrchestrations.ListDurableEntities(IEnumerable1 orchestrations, DurableTaskClient client, Nullable1 timeFrom, Nullable1 timeTill, String[] statuses, HashSet1 hiddenColumns, ILogger log)+MoveNext() in /home/runner/work/DurableFunctionsMonitor/DurableFunctionsMonitor/durablefunctionsmonitor.dotnetisolated.core/Functions/Orchestrations.cs:line 203 at DurableFunctionsMonitor.DotNetIsolated.FilterClauseExtensions.ApplyFilter[T](IEnumerable1 items, FilterClause filter)+MoveNext() in /home/runner/work/DurableFunctionsMonitor/DurableFunctionsMonitor/durablefunctionsmonitor.dotnetisolated.core/Common/FilterClause.cs:line 197 at System.Collections.Generic.EnumerableHelpers.ToArray[T](IEnumerable1 source, Int32& length) at System.Linq.Buffer1..ctor(IEnumerable1 source) at System.Linq.OrderedEnumerable1.GetEnumerator(Int32 minIdx, Int32 maxIdx)+MoveNext() at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty) at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty) at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType) at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType) at Newtonsoft.Json.JsonSerializer.Serialize(JsonWriter jsonWriter, Object value, Type objectType) at Newtonsoft.Json.JsonConvert.SerializeObjectInternal(Object value, Type type, JsonSerializer jsonSerializer) at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, Type type, JsonSerializerSettings settings) at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, JsonSerializerSettings settings) at DurableFunctionsMonitor.DotNetIsolated.Globals.ReturnJson(HttpRequestData req, Object result, Func2 applyThisToJson) in /home/runner/work/DurableFunctionsMonitor/DurableFunctionsMonitor/durablefunctionsmonitor.dotnetisolated.core/Common/Globals.cs:line 127 at Dfm.DotNetIsolatedMsSql.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in /home/runner/work/DurableFunctionsMonitor/DurableFunctionsMonitor/custom-backends/dotnetIsolated-mssql/obj/Release/net8.0/Microsoft.Azure.Functions.Worker.Sdk.Generators/Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator/GeneratedFunctionExecutor.g.cs:line 128 at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13 at DurableFunctionsMonitor.DotNetIsolated.ExtensionMethods.<>c__DisplayClass0_0.<<UseDurableFunctionsMonitor>b__1>d.MoveNext() in /home/runner/work/DurableFunctionsMonitor/DurableFunctionsMonitor/durablefunctionsmonitor.dotnetisolated.core/Common/ExtensionMethods.cs:line 107

mmajcica avatar May 22 '25 06:05 mmajcica

The largest row of the data in the whole DB, judging by SQL is 240356 bytes. This is way lower than 4MB. So, I'm not sure how is that data composed and what else is there, but this does not look as an data issue, but a bug.

mmajcica avatar May 22 '25 08:05 mmajcica