saito-lite-rust icon indicating copy to clipboard operation
saito-lite-rust copied to clipboard

Find a suitable method of profiling nodes to find memory leaks.

Open arpee opened this issue 1 year ago • 2 comments

The primary prod node uses increasingly more memory over time - eventually causing crashes if not restarted.

Image

This is likely a memory leak. To find it we need to be able to profile memory usage to find which objects in the stack are 'bloating'.

Several methods are mentioned here of which 'clinic' looks the most promising.

https://reintech.io/blog/the-best-tools-for-profiling-nodejs-applications

We need to:

  • [ ] find something that can identify memory usage by component (locally)
  • [ ] find a way to implement this in prod if it is not obvious from local profiling where the issue is.

Note: to run nodejs debugging tools agains ts-node - the following syntax is needed: clinic doctor -- node -r ts-node/register start.ts (where "node -r ts-node/register" is added between the regular command and the target script.

arpee avatar Dec 20 '23 21:12 arpee

There are two options I have found

v8-profiler-next

Image

This approach means updating the GitHub repository with the necessary (very minimal) profiling code and deploying it without requiring it to stop or causing downtime. The profiling can be configured to run for a period after which the data is dumped. This data can then be analysed at using Chrome Dev Tools

  • Advantage Minimal to Zero downtime

2. Clinic.js

Image

This approach requires us to stop the application on Prod and run it with Clinic JS. Application crashing does not provide any dump data, so it needs to be manually stopped for profiling to be completed

Advantage Detailed User interface and graphs to help with analysing

davikstone2 avatar Mar 12 '24 14:03 davikstone2

@davikstone2 - with the first option:

  • can we see where in the code the objects listed in the v8 profiler are?
  • can you set out how we trigger the profiling on the node? We run a command?

arpee avatar Mar 14 '24 04:03 arpee