HangfireIO
Could not convert string to DateTime: 27.03.2023 07:30:00. Path 'CreatedAt', line 1, position 100.
We have the following setup:
2 Windows Server 2022 hosting a asp.net core site 2 Windows Server 2022 hosting two .net core windows services
each of the three applications is redundant on its two servers.
sporadically we get the following errors:
Newtonsoft.Json.JsonReaderException
Could not convert string to DateTime: 27.03.2023 07:30:00. Path 'CreatedAt', line 1, position 100.
Newtonsoft.Json.JsonReaderException: Could not convert string to DateTime: 27.03.2023 07:30:00. Path 'CreatedAt', line 1, position 100.
at Newtonsoft.Json.JsonReader.ReadDateTimeString(String s)
at Newtonsoft.Json.JsonTextReader.FinishReadQuotedStringValue(ReadType readType)
at Newtonsoft.Json.JsonTextReader.ReadStringValue(ReadType readType)
at Newtonsoft.Json.JsonTextReader.ReadAsDateTime()
at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
at Hangfire.Batches.BatchJobStorageExtensions.GetBatchState(IStorageConnection connection, String batchId)
at Hangfire.Batches.BatchJobSupportAttribute.OnStateElection(ElectStateContext context)
at Hangfire.States.StateMachine.InvokeOnStateElection(Tuple`2 x)
the error seems to appear only on jobs executed by the webservers, logging shows that it happens on both servers. originally we used sql server storage, but we moved to redis and the issues go on.
interestingly the format of the CreatedAt field in the jobs is sometimes 1679880723201 and sometimes 2023-03-27T07:30:00.4875736Z but never the format that shows up in the exception. but both formats sometimes work and throw the above exception from time to time.
also worth mentioning is that the failing jobs are part of a batch.
Could this error be caused by the incompatibility in the format of the date/time value between the client and the server? It's a good idea to check that first. Because the format "27.03.2023 07:30:00" may not be recognized by the server's CultureInfo or DateTimeFormatInfo settings. I suggest you to check the JSON configurations as well. Are you sure the jsonserializer is configured correctly to handle DateTime values correctly? In summary, you should check the DateTime format in the JSON document and make sure that the server's CultureInfo or DateTimeFormatInfo matches the expected format.
It was my first guess, that the locale differs on the servers, but thats not the case. also, the problem appears while picking up the job. in the storage there is nowhere the format "27.03.2023 07:30:00", so it seems like, its picking it from the storage, converting it to the format and then tries to parse it. also, all 6 application instances are client and server, regarding hangfire.
we dont have any culture specific configuration stored in json files.
we are not using newtonsoft ourselves, afaik its only used by hangfire and installed as a dependency, so all configurations should happen there, right?
After executing the following PowerShell commands in a elevated prompt and with the service/apppool account on all 4 servers and restarting them it seems to work now.
Set-Culture en-US
Set-WinSystemLocale -SystemLocale en-US
Set-WinUILanguageOverride -Language en-US
Set-WinUserLanguageList en-US -Force
Set-WinHomeLocation -GeoId 244
Trying to change these settings via the UI did not work.
The most strange here is how the following value was passed to the database:
Could not convert string to DateTime: 27.03.2023 07:30:00
Batch states are serialized with Newtonsoft.Json with custom settings that don't touch date serialisation formats, which can't be modified externally other than by setting the JsonConvert.DefaultSettings globally for the application. Newtonsoft.Json's default strategy for writing dates is ISO 8601 (2023-03-27T07:30:00.4875736Z-like values), which can't explicitly depend on the CultureInfo.CurrentCulture values since version 4.5.1 (Hangfire.Core supports minimum 5.0.1 version). The default culture for Newtonsoft.Json is CultureInfo.InvariantCulture for more than 10 years, so totally don't understand how that value can be written there.
Unfortunately I was a little too fast with closing this issue, it had stopped for like 3 hours and then came up again. 16 failed jobs in the past 20 hours (of ~8k scheduled jobs). If I check it in redis directly, the format is ISO 8601. so there must be a layer between GetBatchState and the DAL that deserializes from redis and then serializes again to string?
Furthermore I can now confirm, that it seems to be an issue with deserialising. Problems only occur if the current day is > 12. It seems like its mixing up day and month somewhere.
Hi, I found this exact problem as well. Tried to search in the codebase anything that could affect this but no luck. I've crated a post discuss.hangfire.io and Stackoverflow. Thanks
Problems only occur if the current day is > 12. It seems like its mixing up day and month somewhere.
It is because the US date format wich starts with the Month
https://zebulon-delusion6.github.io/OI3vHBfFM3MHTddNZ1SemaUw9k5pzjEuvNgQtQlBRhdJ0nt7mpmXWW/
Note this perspective
