reverse-proxy
reverse-proxy copied to clipboard
microsoft
Proxy SignalR -- document how
Using Yarp proxy SignalR, I frequently encounter connection failures, about 10 requests out of 3 failed, but ocelot did not encounter this situation
Below is my routing configuration
{
"Routes": {
"SignalRRoute": {
"ClusterId": "SignalRCluster",
"CorsPolicy": "CorsPolicy",
"AuthorizationPolicy": "JwtPolicy",
"Match": {
"Path": "/BaizeHub/{*any}"
},
"Transforms": [
{
"PathRemovePrefix": "/BaizeHub"
},
{
"RequestHeadersCopy": "true"
},
{
"RequestHeaderOriginalHost": "true"
},
{
"RequestHeader": "Upgrade",
"Set": "WebSocket"
},
{
"RequestHeader": "Connection",
"Set": "Upgrade"
},
{
"X-Forwarded": "Set",
"For": "Append",
"Proto": "Append",
"Prefix": "Append",
"HeaderPrefix": "X-Forwarded-"
}
]
}
},
"Clusters": {
"SignalRCluster": {
"LoadBalancingPolicy": "PowerOfTwoChoices",
"Destinations": {
"Message/Destination1": {
"Address": "https://172.30.50.13:5400"
},
"Message/Destination2": {
"Address": "https://172.30.50.12:5400"
}
},
"HttpClient": {
"DangerousAcceptAnyServerCertificate": true
},
"HttpRequest": {
"Version": "1.1",
"Timeout": "00:10:00"
},
"HealthCheck": {
"Active": {
"Enabled": "true",
"Interval": "00:00:10",
"Timeout": "00:00:10",
"Policy": "ConsecutiveFailures",
"Path": "/Health"
},
"Passive": {
"Enabled": true,
"Policy": "TransportFailureRate",
"ReactivationPeriod": "00:00:10"
}
}
}
}
}
- Yarp.ReverseProxy 1.0.0-preview.12.21328.2
- Linux
Triage: What are the failures you're getting? Can you share server, client and YARP logs?
Triage: What are the failures you're getting? Can you share server, client and YARP logs?
Yarp logs: 2021-07-21.log
Client:
1.Success
2.Fail
What does the Notify request status translate to? Aborted, failed, etc.?
2021-07-21 09:12:22.9642 [Info] [0] Yarp.ReverseProxy.Forwarder.HttpForwarder+Log.Proxying : Proxying to https://172.30.50.13:5400/Notify?id=llr5GncW9Me-mzNuWzMJFQ&access_token=***
2021-07-21 09:12:22.9993 [Info] [0] Yarp.ReverseProxy.Forwarder.HttpForwarder+Log.Proxying : Proxying to https://172.30.50.12:5400/Notify?id=llr5GncW9Me-mzNuWzMJFQ
2021-07-21 09:12:28.8406 [Info] [0] Yarp.ReverseProxy.Forwarder.HttpForwarder+Log.ErrorProxying : ResponseBodyCanceled: Copying the response body was canceled.
I'm suspicious that there are two calls to Notify, one with the access_token and one without. I can't tell from here which was cancelled.
That cancellation is the only failure reported in the YARP logs. You should only be getting that cancellation error if the client disconnected. If you enable full Debug Microsoft logs for the YARP process that may give more useful details.
We should document how to make SignalR work with YARP on the main docs site after we resolve this.
What does the Notify request status translate to? Aborted, failed, etc.?
2021-07-21 09:12:22.9642 [Info] [0] Yarp.ReverseProxy.Forwarder.HttpForwarder+Log.Proxying : Proxying to https://172.30.50.13:5400/Notify?id=llr5GncW9Me-mzNuWzMJFQ&access_token=*** 2021-07-21 09:12:22.9993 [Info] [0] Yarp.ReverseProxy.Forwarder.HttpForwarder+Log.Proxying : Proxying to https://172.30.50.12:5400/Notify?id=llr5GncW9Me-mzNuWzMJFQ 2021-07-21 09:12:28.8406 [Info] [0] Yarp.ReverseProxy.Forwarder.HttpForwarder+Log.ErrorProxying : ResponseBodyCanceled: Copying the response body was canceled.I'm suspicious that there are two calls to Notify, one with the access_token and one without. I can't tell from here which was cancelled.
That cancellation is the only failure reported in the YARP logs. You should only be getting that cancellation error if the client disconnected. If you enable full Debug Microsoft logs for the YARP process that may give more useful details.
The full logs: 2021-07-21.log
You need sticky sessions to make SignalR work on multiple servers
Maybe I know the reason, is there an IpHash LoadBalancingPolicy?
Sticky sessions also solves this problem. you should try using it.
Let me have a try,thank you!
Sticky sessions also solves this problem. you should try using it.
"SessionAffinity": {
"Enabled": true,
"Policy": "CustomHeader",
"FailurePolicy": "Redistribute",
"AffinityKeyName": "Cookie"
}
I configured it like this, but it doesn't work
Try:
"SessionAffinity": {
"Enabled": true,
"AffinityKeyName": "Yarp.Session"
}
@zqlovejyc - please leave this open as we are using it to track improving the documentation for this area.
@Lukejkw what issues are you hitting? Have you tried the session affinity setting shown above?
@Lukejkw what issues are you hitting? Have you tried the session affinity setting shown above?
No issue per se.
I'm evaluating YARP for an internal gateway where SignalR will be used for real-time communication. I was looking for detailed documentation on if this would work - particularly in a distributed application with multiple SignalR server instances.
There's not a lot of detail to give, enabling session affinity is the main requirement. WebSockets, Long Polling, and SSE should just work.
Fair enough.
Are there any implications to the session affinity requirement if you are running multiple gateway and downstream signalr services instances?
Or does the cookie/header contain the necessary information to route to the correct downstream service regardless of which gateway instance is hit?
The affinity cookie is encrypted so you'll have to have shared data protection keys across instances. See https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/web-farm?view=aspnetcore-6.0#data-protection
Otherwise, as long as the gateways all have the same config it should work. The destination id is used as the session affinity key, so that should be stable across instances.
Great to know. Thanks.
Is the destination ID manually defined or determined dynamically based on the downstream service instance responding to the request?
In a kubernetes environment you would forward traffic from the gateway to a load balancer. So you would define one downstream destination. Would this pose a problem for session affinity and by extension SignalR?
Is the destination ID manually defined or determined dynamically based on the downstream service instance responding to the request?
That depends on your config provider. For the Json config it's manual. Not sure about the Kubernetes provider. @MihaZupan?
In a kubernetes environment you would forward traffic from the gateway to a load balancer. So you would define one downstream destination. Would this pose a problem for session affinity and by extension SignalR?
You don't need session affinity if you only have one destination, it's the load balancer that needs to implement affinity. Not sure why you have both a gateway and a load balancer though, why not combine them?
I’m referring to the kubernetes service which acts as a load balancer between pods. Session affinity can be set there too so that might be the ticket.
Thanks so much for the help.