Clicking 'Trigger Now' of recurring job gets a '422' return

[KevinTyGu]

trafficstars

Hi, I've built a recurring job and tried to run it manually by clicking 'Trigger Now'. The page did not refresh, and I found a 422 return for the POST request of 'trigger'.

I'm using PostgresDB for my project. My configurations are as follows:

I've also noticed this issue#1564, in which the author claimed to succeed with a default SQL Server Storage configuration. I'm unsure if my Postgres configuration or something else causes the issue. Can anyone give some ideas?

[odinserj]

Thanks for all the details. These are the lines where 422 status code is set for the Trigger button, so perhaps there are problems with decoding the job identifiers.

https://github.com/HangfireIO/Hangfire/blob/d51e2c54dc49b28b3f340bb19d5fdcc394bb1b5d/src/Hangfire.Core/Dashboard/BatchCommandDispatcher.cs#L47-L52

Can you show me full HTTP request details from the browser?

[KevinTyGu]

Hi @odinserj , Thanks for the reply. My request payload is as follows:

And the headers are:

cookies:

initiators:

[odinserj]

Unfortunately can't reproduce this, looks like there are problems with reading the form values, since it happens before even going to the job storage in the lines referenced earlier. Can you try to set a breakpoint in the BatchCommandDispatcher.Dispatch method and step over some lines to confirm that the GetFormValuesAsync method returns an empty collection? The class is internal, but source link is enabled for all the recent versions in Hangfire.

[KevinTyGu]

Hi @odinserj , I got the following return of GetFormValuesAsync. The request form is empty, and the code goes to 422 here. My Hangfire version is 1.8.17.

Expanded request:

context.Request = AspNetCoreDashboardRequest
 LocalIpAddress = {string} "::1"
 Method = {string} "POST"
 Path = {string} "/recurring/trigger"
 PathBase = {string} "/hangfire"
 RemoteIpAddress = {string} "::1"
 _context = {DefaultHttpContext} POST https://localhost:7191/hangfire/recurring/trigger HTTP/2 application/x-www-form-urlencoded; charset=UTF-8 200 OK
  Connection = {DefaultConnectionInfo} Id = 0HNCSGJ22QCRD, Remote = [::1]:65355, Local = [::1]:7191
  Endpoint = {Endpoint} null
  Features = {HttpContext.HttpContextFeatureDebugView} Count = 30
  Items = {ItemsDictionary} Count = 0
  Request = {DefaultHttpRequest} POST https://localhost:7191/hangfire/recurring/trigger HTTP/2 application/x-www-form-urlencoded; charset=UTF-8
   ContentLength = {long} 37
   ContentType = {string} "application/x-www-form-urlencoded; charset=UTF-8"
   Cookies = {RequestCookieCollection} Count = 2
   Form = FormCollection
     Count = {int} 0
     Files = {FormFileCollection} Count = 0
     Keys = {Dictionary<string, StringValues>.KeyCollection} Count = 0
     Store = {Dictionary<string, StringValues>} Count = 0
     _files = {IFormFileCollection} null
      Empty = FormCollection
      EmptyFiles = {FormFileCollection} Count = 0
      EmptyIEnumerator = FormCollection.Enumerator
      EmptyIEnumeratorType = FormCollection.Enumerator
      EmptyKeys = {string[]} string[0]
     Empty = {string} "Enumeration yielded no results"
   HasFormContentType = {bool} true
   Headers = {HttpRequestHeaders} Count = 19
   Host = {HostString} localhost:7191
   IsHttps = {bool} true
   Method = {string} "POST"
   Path = {PathString} /recurring/trigger
   PathBase = {PathString} /hangfire
   Protocol = {string} "HTTP/2"
   Query = {QueryCollection} Count = 0
   QueryString = {QueryString} 
   RouteValues = {RouteValueDictionary} Count = 0
   Scheme = {string} "https"
  RequestAborted = {CancellationToken} IsCancellationRequested = false
  RequestServices = {ServiceProviderEngineScope} ServiceDescriptors = 497, IsScope = true
  Response = {DefaultHttpResponse} 200 OK
  Session = {ISession} null
  TraceIdentifier = {string} "0HNCSGJ22QCRD:0000000F"
  User = {ClaimsPrincipal} IsAuthenticated = false
  WebSockets = {DefaultWebSocketManager} IsWebSocketRequest = false

[odinserj]

Great, so we have a request with x-www-form-urlencoded parameters, and an empty FormCollection in ASP.NET Core, and ContentType property is set to application/x-www-form-urlencoded; charset=UTF-8, so there should be no issues.

Request = {DefaultHttpRequest} POST https://localhost:7191/hangfire/recurring/trigger HTTP/2 application/x-www-form-urlencoded; charset=UTF-8
   ContentLength = {long} 37
   ContentType = {string} "application/x-www-form-urlencoded; charset=UTF-8"
   Cookies = {RequestCookieCollection} Count = 2
   Form = FormCollection
     Count = {int} 0
     Files = {FormFileCollection} Count = 0
     Keys = {Dictionary<string, StringValues>.KeyCollection} Count = 0
     Store = {Dictionary<string, StringValues>} Count = 0
     _files = {IFormFileCollection} null
      Empty = FormCollection
      EmptyFiles = {FormFileCollection} Count = 0
      EmptyIEnumerator = FormCollection.Enumerator
      EmptyIEnumeratorType = FormCollection.Enumerator
      EmptyKeys = {string[]} string[0]
     Empty = {string} "Enumeration yielded no results"

Please try digging into Hangfire's AspNetCoreDashboardRequest.GetFormValuesAsync method, then into ASP.NET Core's ReadFormAsync method, and then into the FormFeature's InnerReadFormAsync method, it should go into the following line (please show me if it isn't):

And then from the FormPipeReader.ReadFormAsync it should go to the ParseFormValues method:

And finally to the ParseFormValuesFast (more likely) or ParseFormValuesSlow (less likely), where decodedKey should be jobs[], and decodedValue should be FailedGtsDbRequest...:

Please show me screenshots of any differences in this flow – when another if branch is chosen, other methods are getting called, etc. – we should understand where ASP.NET Core is choosing a wrong path.

[odinserj]

These are my results:

[KevinTyGu]

Hi @odinserj, I've tried your debugging methods in Rider on Mac. However, I could only debug the decompiled code since Rider on Mac does not support a PDB symbol server. And I couldn't reach ParseFormValues, where ParseFormValuesFast or ParseFormValuesSlow reside.

Also, seems I could not view the variable values. Do I have to use the official source code for external debugging?

[odinserj]

Hm, this class comes from the Microsoft.AspNetCore.WebUtilities package that uses Source Link to provide source code since version 2.1.0 released in 2018. What .NET version you are using?

[KevinTyGu]

Hm, this class comes from the Microsoft.AspNetCore.WebUtilities package that uses Source Link to provide source code since version 2.1.0 released in 2018. What .NET version you are using?

I'm using SDK 8.0.300, and the AspNetCore should be 8.0.5.

[KevinTyGu]

Hi @odinserj, could you please send me the versions of your environment? I'll try it locally.

[odinserj]

Hm, your project is new enough to avoid that problem with source link. My project was based on SDK 8.0.410. What if we include Microsoft.AspNetCore.WebUtilities package of its latest version explicitly in the project? Does it solve the problem?

[KevinTyGu]

Hi @odinserj, sorry I was unable to succeed with a version of these. And debugging on JetBrains Rider on Mac always returns 'An IL variable is not available at the current native IP' for all variables.

[KevinTyGu]

Hi @odinserj, it seems the issue has become severe now that it kinda affects our production environment, where 'requeue jobs' also returns a 422 error on our production Dashboard. The Hangfire version is 1.8.17. It seems there is a compatibility issue with the UI or something else?

However, the 'Requeue' button within a single job still works fine.

[odinserj]

Unfortunately I don't understand what happens here – we have a proper HTTP request with the required type and form data, we call ASP.NET Core's method to read this form, and get nothing in return, so Dashboard UI can't process the request, since no data is passed from ASP.NET Core's side. The only idea that just came to me is that other middleware registered in the application prevents from doing that for some reason, and we can try putting the UseHangfireDashboard method call before registering that middleware. Can this be the case?