Viewing historical data for cluster results in exception during JSON serialization: exception.stacktrace: System.ArgumentException: .NET number values such as positive and negative infinity cannot be written as valid JSON.

[ElanHasson]

I am unable to access the silo details and tracing shows the following exception:

Using latest version of Dashboard and Orleans 3.6.0 and Dashboard 3.6.1. .NET 6.

Repo is here: https://github.com/ElanHasson/web-scheduler/tree/main/Source/WebScheduler.Server

Dashboard is: https://scheduler.nullreference.io/

web-scheduler-server 2022-01-21T06:53:05.307880179Z Activity.ActivitySourceName: OpenTelemetry.Instrumentation.AspNetCore
web-scheduler-server 2022-01-21T06:53:05.307900689Z Activity.DisplayName: /HistoricalStats/10.244.12.103:11111@380443774
web-scheduler-server 2022-01-21T06:53:05.307906161Z Activity.Kind:        Server
web-scheduler-server 2022-01-21T06:53:05.307912474Z Activity.StartTime:   2022-01-21T06:53:05.2795081Z
web-scheduler-server 2022-01-21T06:53:05.307918212Z Activity.Duration:    00:00:00.0276737
web-scheduler-server 2022-01-21T06:53:05.307929945Z Activity.TagObjects:
web-scheduler-server 2022-01-21T06:53:05.307967454Z     http.host: scheduler.nullreference.io
web-scheduler-server 2022-01-21T06:53:05.308052595Z     http.method: GET
web-scheduler-server 2022-01-21T06:53:05.308126847Z     http.target: /HistoricalStats/10.244.12.103:11111@380443774
web-scheduler-server 2022-01-21T06:53:05.308294285Z     http.url: http://scheduler.nullreference.io/HistoricalStats/10.244.12.103:11111@380443774
web-scheduler-server 2022-01-21T06:53:05.308313987Z     http.user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36 Edg/97.0.1072.62
web-scheduler-server 2022-01-21T06:53:05.308385125Z     http.flavor: 1.1
web-scheduler-server 2022-01-21T06:53:05.308467969Z     http.scheme: http
web-scheduler-server 2022-01-21T06:53:05.308527617Z     http.client_ip: ::ffff:10.244.3.3
web-scheduler-server 2022-01-21T06:53:05.308627828Z     http.request_content_length: 
web-scheduler-server 2022-01-21T06:53:05.308697690Z     http.request_content_type: application/json
web-scheduler-server 2022-01-21T06:53:05.308774614Z     otel.status_code: ERROR
web-scheduler-server 2022-01-21T06:53:05.309076512Z     otel.status_description: .NET number values such as positive and negative infinity cannot be written as valid JSON.
web-scheduler-server 2022-01-21T06:53:05.309082918Z     http.status_code: 500
web-scheduler-server 2022-01-21T06:53:05.309086851Z     http.response_content_length: 0
web-scheduler-server 2022-01-21T06:53:05.309089336Z     http.response_content_type: 
web-scheduler-server 2022-01-21T06:53:05.309301406Z Activity.Events:
web-scheduler-server 2022-01-21T06:53:05.309738207Z     exception [1/21/2022 6:53:05 AM +00:00]
web-scheduler-server 2022-01-21T06:53:05.309801109Z         exception.type: System.ArgumentException
web-scheduler-server 2022-01-21T06:53:05.310034488Z         exception.stacktrace: System.ArgumentException: .NET number values such as positive and negative infinity cannot be written as valid JSON.
web-scheduler-server 2022-01-21T06:53:05.310041676Z    at System.Text.Json.ThrowHelper.ThrowArgumentException_ValueNotSupported()
web-scheduler-server 2022-01-21T06:53:05.310045000Z    at System.Text.Json.Utf8JsonWriter.WriteNumberValue(Single value)
web-scheduler-server 2022-01-21T06:53:05.310048220Z    at System.Text.Json.Serialization.Converters.SingleConverter.Write(Utf8JsonWriter writer, Single value, JsonSerializerOptions options)
web-scheduler-server 2022-01-21T06:53:05.310056261Z    at System.Text.Json.Serialization.Converters.NullableConverter`1.Write(Utf8JsonWriter writer, Nullable`1 value, JsonSerializerOptions options)
web-scheduler-server 2022-01-21T06:53:05.310058890Z    at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
web-scheduler-server 2022-01-21T06:53:05.310114512Z    at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer)
web-scheduler-server 2022-01-21T06:53:05.310132660Z    at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
web-scheduler-server 2022-01-21T06:53:05.310137144Z    at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
web-scheduler-server 2022-01-21T06:53:05.310139626Z    at System.Text.Json.Serialization.Converters.ArrayConverter`2.OnWriteResume(Utf8JsonWriter writer, TElement[] array, JsonSerializerOptions options, WriteStack& state)
web-scheduler-server 2022-01-21T06:53:05.310142776Z    at System.Text.Json.Serialization.JsonCollectionConverter`2.OnTryWrite(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state)
web-scheduler-server 2022-01-21T06:53:05.310149983Z    at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
web-scheduler-server 2022-01-21T06:53:05.310200501Z    at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
web-scheduler-server 2022-01-21T06:53:05.310218560Z    at System.Text.Json.JsonSerializer.WriteUsingSerializer[TValue](Utf8JsonWriter writer, TValue& value, JsonTypeInfo jsonTypeInfo)
web-scheduler-server 2022-01-21T06:53:05.310229187Z    at System.Text.Json.JsonSerializer.Serialize[TValue](Utf8JsonWriter writer, TValue value, JsonSerializerOptions options)
web-scheduler-server 2022-01-21T06:53:05.310232101Z    at OrleansDashboard.DashboardMiddleware.WriteJson[T](HttpContext context, T content)
web-scheduler-server 2022-01-21T06:53:05.310235089Z    at OrleansDashboard.DashboardMiddleware.WriteJson[T](HttpContext context, T content)
web-scheduler-server 2022-01-21T06:53:05.310256308Z    at OrleansDashboard.DashboardMiddleware.Invoke(HttpContext context)
web-scheduler-server 2022-01-21T06:53:05.310260769Z    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
web-scheduler-server 2022-01-21T06:53:05.310462457Z         exception.message: .NET number values such as positive and negative infinity cannot be written as valid JSON.
web-scheduler-server 2022-01-21T06:53:05.310565476Z Resource associated with Activity:
web-scheduler-server 2022-01-21T06:53:05.310796066Z     deployment.environment: Development
web-scheduler-server 2022-01-21T06:53:05.310802980Z     host.name: web-scheduler-server-7748456fc6-9txtx
web-scheduler-server 2022-01-21T06:53:05.310824105Z     service.name: WebScheduler.Server
web-scheduler-server 2022-01-21T06:53:05.310904197Z     service.version: 1.0.0.0
web-scheduler-server 2022-01-21T06:53:05.310988044Z     service.instance.id: 4d92a77f-4e87-4105-a56f-02d88b4ed2f0

[richorama]

Thanks for reporting this @ElanHasson I will investigate.

[ElanHasson]

@richorama funny enough, I randomly checked this morning: https://scheduler.nullreference.io/HistoricalStats/10.244.12.103:11111@380443774

but that is a dead silo now :)

[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null]

[ElanHasson]

I think I figured it out. I'm running in a container using gVisor. My guess is CPU/memory metrics are not allowed in this environment.

I saw the ECS provider, so I'll have to do similar if I want those metrics 😂

I do expect not a 500 and to see the page, just without those metrics, i.e. no error.

Will experiment.

[ElanHasson]

Yep as expected gVisor doing me sorry here.

https://scheduler.nullreference.io/OrleansDashboard/#/host/10.244.15.84:11111@381194005

Works fine now.

[ElanHasson]

An update:

Output from container running on gVisor: /proc/meminfo (seems to work fine)

/proc/stat (returns all 0s: https://github.com/google/gvisor/blob/master/pkg/sentry/fs/proc/stat.go#L88-L104)

The issue may be here: https://github.com/dotnet/orleans/blob/main/src/TelemetryConsumers/Orleans.TelemetryConsumers.Linux/LinuxEnvironmentStatistics.cs#L163-L166

[SebastianStehle]

Does it mean we can close this issue?

[richorama]

I think we should keep this open, as the dashboard should fail gracefully in this situation.

[SebastianStehle]

Okay, but the SiloRuntimeStatistics is from Orleans. We either have to loop over all propperties to fix invalid numbers or write a custom Serializer for Single and doubles.

[ElanHasson]

I'm asking @ReubenBond and team on what they'd like the best fix to be on the Orleans side: https://discord.com/channels/333727978460676096/922945034427432980/937707474570608710

Will send a PR for it to them.

[ElanHasson]

Here is the PR For Orleans which will allow CpuLimit to not be set.

https://github.com/dotnet/orleans/pull/7526