SpacetimeDB icon indicating copy to clipboard operation
SpacetimeDB copied to clipboard

Track memory usage for tables in grafana & track energy for persistent database memory

Open joshua-spacetime opened this issue 2 years ago • 3 comments

We should track the persistent memory associated with tables and indexes for energy purposes. This means each table and index needs access to an EnergyMonitor for recording energy use.

If 1 eV is the amount of energy used to store 1 byte on disk for 1 second, then N eV is the amount of energy required to store 1 byte in memory for 1 second for some N. We should pick a suitable N and record, for every insert/delete, the amount of energy used per table, since the previous insert/delete.

As part of this ticket we should also write these energy recordings to prometheus.

joshua-spacetime avatar Dec 13 '23 01:12 joshua-spacetime

100 is not so far from a suitable N (for context)

cloutiertyler avatar Dec 13 '23 02:12 cloutiertyler

We would like to see the number of bytes per table so that we can track which tables have very large rows.

cloutiertyler avatar Apr 11 '24 17:04 cloutiertyler

We would like to see the number of bytes per table so that we can track which tables have very large rows.

It would be very easy to compute a conservative overestimate of this by simply tracking the number of pages per table in the committed state. A precise count is more challenging.

gefjon avatar Apr 11 '24 18:04 gefjon

I split this ticket into https://github.com/orgs/clockworklabs/projects/22/views/38?pane=issue&itemId=47389802 as well

cloutiertyler avatar May 28 '24 18:05 cloutiertyler