make psn-embedded sqlite use a /tmp file instead of an in-memory DB

[tanelpoder]

Currently sqlite uses only memory to keep track of thread samples.

This could take a noticeable amount of memory, especially when sampling all thread states, at high frequency and for a long time.

When initializing a sqlite DB/connection, configure it to use a /tmp file instead of RAM. We could make it conditional, a'la if psn is executed to sample for more than N seconds (60 seconds?), then use a file-backed sqlite DB, otherwise RAM.

[tanelpoder]

Thanks for the quick improvement, looks good!

The original 0x.tools and psn were designed to run even on very old Linux distros like RHEL5/RHEL6 (on python2 without f-strings). It's time to phase out the distant legacy stuff, so I'll probably do a final release that still works on old distros and then merge + release a modern version.

Do you have any feedback re: how you use this tool, where it's valuable and if you've found shortcomings? Helps me to pick what to implement next.... feel free to either open a discussion or just email to [email protected] works too.

[tanelpoder]

Do you have time to approach this from a little different angle - as there are 3-4 factors that determine how much data would be needed for buffering in SQLite before reporting:

1. Sampling duration
2. Sampling frequency
3. Whether -a or -A are used (that capture all "idle" threads too)
4. Which dimensions are captured & reported (how many columns)

Implementing the decision based on 1,2,3 would be relatively easy, Duration * Frequency * AllSamples. The AllSamples could be just 1 by default (expecting only the R+D threads), but if -a or -A option is used, then the factor would be 10 or more.

Of course I don't want to complicate this too much, maybe an easier way is to just use Duration * Frequency and when the expected number of total samples taken is more than X, then switch to SQLite.

Now that I wrote this, seems like unnecessary complexity added. Perhaps a better option is to just add a command line option to force either /tmp or mem storage for special cases (like when someone runs psn just for 5 seconds but in a very memory-constrained environment).

I think one input variable for this decision would be to run psn with a couple of different duration + freqency arguments (in a busy system) and then see what the actual memory usage is.

[tanelpoder]

One example scenario what I think of is that an admin logs in to a server that "seems slow". The 1st thing I do is usually run psn to see what's the "demand" at the app/kernel/thread level in the machine. But if it happens to be a bad memory shortage problem, then psn itself would take a long time. Maybe one option is just to switch the decision point from 60 seconds to 5-10 seconds..