timestored
Large query causes client memory problems
user-rh reported "A typical chart query will be approximately 600k rows with 3 columns (time, price1, price2) and I poll every 10 seconds. This has cause the front end to crash a couple of times due to running out of memory."
It's been a while since we looked at reducing memory usage and increasing performance. It's easy for added features to degrade performance. Let's investigate.
Options we have discussed before:
- Increasing speed and reduce memory usage of rendering pipeline (nice part is that increases here benefit everything)
- Object pooling of expensive objects.
- Smart Table Delta
- InfiniteZoom support
Checking for memory leaks, relpayed multiple days data in 20 minute window chunks:
Client memory went up and down as expected.
Tracking objects, shows no long lived objects.
Overall memory of Chrome process went up and down as expected.
THis does still leave the possibility of an issue over time, e.g. maybe a timer? As I was playing back at speed.
On kdb subscription I could cause th eserver memory to spike due to previously found issue: https://github.com/micronaut-projects/micronaut-core/issues/8984
Once the backlog cleared it was fine:
However the client side memory still returned to normal. I then tried polling in case it was polling specific, and again it returned to normal. Tried JS heap snapshots and tracing. Can't find anything long lived.
user-rh reported "crash" then stack trace with:
com.sqldashboards.webby.WebSocketServer$DashboardEngine lambda$new$0
WARNING: Subscriber for admin seems to be a slow subscriber.
com.sqldashboards.dashy.QueryEngine2$1 run
SEVERE: big loopy scheduled problems
io.micronaut.websocket.exceptions.WebSocketSessionException: Session closed
at io.micronaut.http.netty.websocket.NettyWebSocketSession.sendAsync(NettyWebSocketSession.java:166)
at io.micronaut.websocket.WebSocketSession.sendAsync(WebSocketSession.java:154)
I have not been able to recreate this scenario. When a subscriber falls behind they should be slowed (this was), then stopped. This wasn't. I tried force closing the browser and that still didn't give the error in the logs.