suave
suave copied to clipboard
SuaveIO
EventSource resource exhaustion
If the client doesn't properly close the EventSource channel, this is what happens after a while:
[W] 2015-11-06T18:13:40.9217220Z: tcp request processing failed [Suave.Tcp.tcpIpServer.job] exn:
System.InvalidOperationException: Stack empty.
at System.ThrowHelper.ThrowInvalidOperationException (ExceptionResource resource) in <filename unknown>:line 0
at System.Collections.Generic.Stack`1[T].Pop () in <filename unknown>:line 0
at Suave.Sockets.BufferManager.PopBuffer (Microsoft.FSharp.Core.FSharpOption`1 context) in <filename unknown>:line 0
at [email protected] (Microsoft.FSharp.Core.Unit unitVar) in <filename unknown>:line 0
at Microsoft.FSharp.Control.AsyncBuilderImpl+callA@805[b,a].Invoke (Microsoft.FSharp.Control.AsyncParams`1 args) in <filename unknown>:line 0
[W] 2015-11-06T18:13:41.0627360Z: tcp request processing failed [Suave.Tcp.tcpIpServer.job] exn:
System.InvalidOperationException: Stack empty.
at System.ThrowHelper.ThrowInvalidOperationException (ExceptionResource resource) in <filename unknown>:line 0
at System.Collections.Generic.Stack`1[T].Pop () in <filename unknown>:line 0
at Suave.Sockets.BufferManager.PopBuffer (Microsoft.FSharp.Core.FSharpOption`1 context) in <filename unknown>:line 0
at [email protected] (Microsoft.FSharp.Core.Unit unitVar) in <filename unknown>:line 0
at Microsoft.FSharp.Control.AsyncBuilderImpl+callA@805[b,a].Invoke (Microsoft.FSharp.Control.AsyncParams`1 args) in <filename unknown>:line 0
This is because there's no NACK facility in async; should either mimik it with async.Choose with a timeout, so that the socket monad gets 'polled' by having its bind invoked and thereby checking whether the socket is closed.
Hi, is there any known workaround for this? I got web running on Suave (in Docker container in AWS) and after about 2 days I got exactly this error. Redeploy of container seems to be the only way, but that is definitely something you do not want to do on daily basis.
You have time out the connection. Search gitter.im for Async.Choose for an example. Sorry about the issue; I have it on my TODO to fix.
That is ok, web is still in preparing phase (and will be for next few days), but having few more projects in mind, this should not happen. Looking forward to see you fixed it on future release.
Has this been fixed or should we still expect to restart Suave on daily basis?
It's technically not a bug since the TCP connection is open. We're cycling our EventSource sockets every thirty seconds over here... How do you envision fixing this @slav?
I thought about this once, even coded a connection manager to disconnect connections with no activity. I think it makes sense to have such a connection manager and track time spent per request and similar stats.
This is what we have right now; I'd love some feedback and input and perhaps we can create a good component to place in the appropriate module? Obviously this is different because we use Hopac everywhere internally...
let private checkAndDispose (d : #IDisposable) result =
async {
let! r = result
match r with
| Choice1Of2 a ->
return Choice1Of2 a
| Choice2Of2 e ->
d.Dispose()
return Choice2Of2 e
}
let private resettingTimeOut timeSpan (d : #IDisposable) =
let resetCh = Ch()
let rec inner resetCh =
Alt.choose [
Ch.take resetCh |> Alt.afterJob (fun () -> inner resetCh)
timeOut timeSpan |> Alt.afterFun (fun _ -> d.Dispose())
]
start (inner resetCh)
resetCh
// TODO: this currently disposes the stream after 30 seconds staying alive forever
// This could be replaced by a heartbeat mechanism of sending a raw ':\n' over the
// stream which the eventstream specification says the client should always ignore
let inline notificationStreamToWebPart subscribeFun : Connection -> SocketOp<_> =
fun conn ->
socket {
let! (notificationStream : NotificationStream<_>) =
subscribeFun
|> Hopac.Job.map Choice1Of2
|> Job.toAsync
let reset =
resettingTimeOut (TimeSpan.FromSeconds 30.) notificationStream
let guard a = checkAndDispose notificationStream a
return!
Hopac.Stream.foldJob
(fun xxxx x ->
x |> Json.serialize |> Json.format
|> EventSource.Message.create (Guid.NewGuid().ToString())
|> EventSource.send conn
|> SocketOp.map (fun _ -> conn)
|> guard
|> Job.fromAsync
|> Job.bind (fun r -> job { do! Ch.send reset ()
return r }))
(Choice1Of2 conn)
notificationStream.stream
|> Job.toAsync
}