swagger-stats eating our server memory

[MohammedAlasaad]

Hi, I am using swagger-stats with our Loopback 4 and its nice to have and track all errors and stats. but the issue we dont sending the data anywhere and its feels its gettings bigger with millions requests and cpu jump high is there a way to reset the data there every 30 min or something? we dont need to send data and we just need to monitor requests and fix errors if there.

Thanks

[sv2]

Do you by any chance have non-repeating URL patterns that could accumulate over time ? Such as GUID in path, not defined as a URL parameter in routes ? If yes, swagger-stats may try to collect stats on each such URL and stats can grow. One way to prohibit it is to set option swaggerOnly: true. In this case, only APIs defined in your OpenAPI spec will be monitored. If this does not work, then yes, can consider to add stats cleanup method that can be called periodically.

[MohammedAlasaad]

Hi, Thanks for answer, we have only Loopback 4 and swagger-stats there on the server. so it takes that requets. no other requets there and its same on openapi.json I think the issue maybe coz we dont sending data to anywhere and we dont need. we just need to monitor and check things with swagger-stats thats good but we have many requets and when its so high the cpu and ram get really high too then we need to reload LB4 so swagger-stats reload as well. and then it get down again.

[sv2]

Could you try to run your app for some time with swagger-stats debug enabled, and share log ? To enable debug, set env variable DEBUG=sws:* Thanks !

[MohammedAlasaad]

Hi Where I would get the log file ? I am using it like middleware app.use(swStats.getMiddleware({}));

[sv2]

When DEBUG=sws:* is set, swagger-stats logs will be written to stdout ... you can redirect them to file, and then run your app for several minutes when it processes traffic ... if you could then attach this file, this would be helpful.

[smashah]

My express app takes in requests with relatively large JSON. When a request errors out it gets logged in the last errors section with the whole JSON object. Usually the app runs at around 128mb but with swagger stats it continually grows. I think it's because it saves the last 100 errors. If this can be managed via the config I'm sure I can get the memory usage under control.

In the meantime I will attempt to sanitize the JSON in onResponseFinish

[smashah]

Update: I think I have found the source (to at least my) memory leak issues:

https://github.com/slanatech/swagger-stats/blob/9291d6e962cbc5871132317fb0e9bcb0f5c7e13c/lib/swsProcessor.js#L383:L392

My API ingests relatively large request bodies so when those requests end up in the list of errors or longest requests then the large JSON bodies get stuck in memory.

I thought that if I sanitized rrr in onResponseFinish then it would solve my problems and at first, it did seem like it did (because the sanitized bodies show up in the dashboard), however, due to the order of execution (onResponseFinish is executed last) any memory leaks in this.longestRequests.processReqResData(rrr); or this.lastErrors.processReqResData(rrr); are preserved.

From the memory dump, it looks like the unsanitized request bodies get leaked/replicated during the lastError and longestRequests processing (i think it gets stuck in a timer somewhere)

The solution to this is to add a sanitizeRRR configuration (similar to onResponseFinish) where I minimize/sanitize the rrr before it gets to lastErrors or longestrequests.

For the meantime, I have fixed this issue for myself by changing the capacity of swsLongestRequests from 100 to 2 and this.last_errors.length > 100 to this.last_errors.length > 10 in swsLastErrors

Before, memory was pinned to up to 1gb on heroku, now it is maintained under 50mb

[sv2]

Thank you ! Will check what could be cause of this

[smashah]

Hi @sv2

Would really love to reimplement an un-patched version of the library in my project.

I think the next steps to resolve this issue is to:

• [ ] Implement an option to sanitize RRR before it gets caught up in a loop/timer - preSanitizeRRR
• [ ] Implement an option to limit the size of last_error array - maxLastErrors
• [ ] Implement an option to limit the size of longest_request array - maxLongestRequests

[smashah]

@sv2 Is this something you would consider implementing or publishing quickly if I made a PR?

[sv2]

Sure !