pgcat icon indicating copy to clipboard operation
pgcat copied to clipboard

[work in progress] reporting query result statistics

Open dat2 opened this issue 2 years ago • 0 comments

in order to discover what queries are cacheable, we need to know roughly how many times we get the same result for a given query. this implements that by hashing a specific query text, hashing the results we get back, and storing some metadata such as first_seen, last_seen, and count. we can compute a rough TTL for cacheable queries using last_seen - first_seen to see how often query results change (for the exact same query text).

on our databases, we have some tables that see thousands of writes / week, but billions of reads / week, so we're trying to estimate how long we can cache those rows to try and take the load off for those billions of reads.

i'm able to make this work with

  • LOG_LEVEL=debug cargo run -- pgcat.minimal.toml
  • PGPASSWORD=postgres psql -p 6432 -h localhost -d pgml -U postgres -w -c 'select 1'
  • PGPASSWORD=postgres psql -p 6432 -h localhost -d pgml -U postgres -w -c 'select 2'
  • PGPASSWORD=postgres psql -p 6432 -h localhost -d pgml -U postgres -w -c 'select 3'
  • wait some time
  • PGPASSWORD=postgres psql -p 6432 -h localhost -d pgml -U postgres -w -c 'select 1'
  • PGPASSWORD=pgcat psql -p 6432 -h localhost -d pgcat -U pgcat -w -c 'show query_result_stats'
 database |   user   | normalized |     fingerprint     |                            query_hash                            |                           result_hash                            | count |           first_seen           |           last_seen            |    duration
----------+----------+------------+---------------------+------------------------------------------------------------------+------------------------------------------------------------------+-------+--------------------------------+--------------------------------+-----------------
 pgml     | postgres | select $1  | 5836069208177285818 | 7c10e523b62b91c18b1583477de02a827c10e523b62b91c18b1583477de02a82 | 53d70edc5bb4891237a956e17423bfe153d70edc5bb4891237a956e17423bfe1 |     5 | 2023-08-11 13:39:48.633979 UTC | 2023-08-11 13:40:08.208047 UTC | 0d 00:00:19.574
 pgml     | postgres | select $1  | 5836069208177285818 | 3c9c5d447eb1ce24bc442b78cc88f93c3c9c5d447eb1ce24bc442b78cc88f93c | bd131b4cebad41e69e9d8d4f0cda706bd131b4cebad41e69e9d8d4f0cda706   |     1 | 2023-08-11 13:40:04.734028 UTC | 2023-08-11 13:40:04.734029 UTC | 0d 00:00:00.000
 pgml     | postgres | select $1  | 5836069208177285818 | 549b8517156cb240c57f52e7f2f2fa3e549b8517156cb240c57f52e7f2f2fa3e | e14fcf3615636eecb874f0d2f6ca287ee14fcf3615636eecb874f0d2f6ca287e |     1 | 2023-08-11 13:40:05.872017 UTC | 2023-08-11 13:40:05.872018 UTC | 0d 00:00:00.000
(3 rows)

dat2 avatar Jul 25 '23 16:07 dat2