[work in progress] reporting query result statistics #525

dat2 · 2023-07-25T16:58:15Z

in order to discover what queries are cacheable, we need to know roughly how many times we get the same result for a given query. this implements that by hashing a specific query text, hashing the results we get back, and storing some metadata such as first_seen, last_seen, and count. we can compute a rough TTL for cacheable queries using last_seen - first_seen to see how often query results change (for the exact same query text).

on our databases, we have some tables that see thousands of writes / week, but billions of reads / week, so we're trying to estimate how long we can cache those rows to try and take the load off for those billions of reads.

i'm able to make this work with

LOG_LEVEL=debug cargo run -- pgcat.minimal.toml
PGPASSWORD=postgres psql -p 6432 -h localhost -d pgml -U postgres -w -c 'select 1'
PGPASSWORD=postgres psql -p 6432 -h localhost -d pgml -U postgres -w -c 'select 2'
PGPASSWORD=postgres psql -p 6432 -h localhost -d pgml -U postgres -w -c 'select 3'
wait some time
PGPASSWORD=postgres psql -p 6432 -h localhost -d pgml -U postgres -w -c 'select 1'
PGPASSWORD=pgcat psql -p 6432 -h localhost -d pgcat -U pgcat -w -c 'show query_result_stats'

 database |   user   | normalized |     fingerprint     |                            query_hash                            |                           result_hash                            | count |           first_seen           |           last_seen            |    duration
----------+----------+------------+---------------------+------------------------------------------------------------------+------------------------------------------------------------------+-------+--------------------------------+--------------------------------+-----------------
 pgml     | postgres | select $1  | 5836069208177285818 | 7c10e523b62b91c18b1583477de02a827c10e523b62b91c18b1583477de02a82 | 53d70edc5bb4891237a956e17423bfe153d70edc5bb4891237a956e17423bfe1 |     5 | 2023-08-11 13:39:48.633979 UTC | 2023-08-11 13:40:08.208047 UTC | 0d 00:00:19.574
 pgml     | postgres | select $1  | 5836069208177285818 | 3c9c5d447eb1ce24bc442b78cc88f93c3c9c5d447eb1ce24bc442b78cc88f93c | bd131b4cebad41e69e9d8d4f0cda706bd131b4cebad41e69e9d8d4f0cda706   |     1 | 2023-08-11 13:40:04.734028 UTC | 2023-08-11 13:40:04.734029 UTC | 0d 00:00:00.000
 pgml     | postgres | select $1  | 5836069208177285818 | 549b8517156cb240c57f52e7f2f2fa3e549b8517156cb240c57f52e7f2f2fa3e | e14fcf3615636eecb874f0d2f6ca287ee14fcf3615636eecb874f0d2f6ca287e |     1 | 2023-08-11 13:40:05.872017 UTC | 2023-08-11 13:40:05.872018 UTC | 0d 00:00:00.000
(3 rows)

src/stats/query_result_stats.rs

zainkabani · 2023-08-14T20:34:03Z

src/client.rs

@@ -1675,6 +1678,20 @@ where
            Some(message) => message,
            None => &self.buffer,
        };
+        // TODO query sample rate from config
+        let should_parse_query = pool.settings.query_parser_enabled
+            && !server.in_transaction()


Can we also make sure it's not a begin statement we're caching or a set statement? Ideally we just check for select from

levkk · 2023-08-21T19:13:49Z

Cargo.toml

@@ -3,6 +3,9 @@ name = "pgcat"
 version = "1.1.1"
 edition = "2021"

+[profile.release]
+debug = true


The executable is gonna be quite large. Not an issue for production imo.

Is this really required, though?

I committed it unnecessarily

It's great for debugging production issues, otherwise the backtraces are pretty slim. This just attaches debug symbols, but still compiles it with all the optimizations.

mdashti

@dat2 Thanks for the PR. This is certainly exciting.

mdashti · 2023-10-13T23:47:52Z

Cargo.toml

@@ -3,6 +3,9 @@ name = "pgcat"
 version = "1.1.1"
 edition = "2021"

+[profile.release]
+debug = true


Is this really required, though?

mdashti · 2023-10-13T23:50:07Z

Cargo.toml

@@ -48,6 +51,8 @@ itertools = "0.10"
 clap = { version = "4.3.1", features = ["derive", "env"] }
 tracing = "0.1.37"
 tracing-subscriber = { version = "0.3.17", features = ["json", "env-filter", "std"]}
+pg_query = "0.7"


If we're adding pg_query, shouldn't we also remove sqlparser (and replace its usages with pg_query)?

yes! feel free to do it. I wont' be able to get to it for a month or so

mdashti · 2023-10-13T23:52:51Z

src/lib.rs

@@ -40,3 +41,10 @@ pub fn format_duration(duration: &chrono::Duration) -> String {
        days, hours, minutes, seconds, milliseconds
    )
 }
+
+pub fn hash_string(hash: &[u8]) -> String {
+    // TODO assert length == 256


Suggested change

// TODO assert length == 256

debug_assert_eq!(hash.len(), 256)

Is that enough?

mdashti · 2023-10-13T23:58:35Z

src/query.rs

+
+    pub fn is_select(&self) -> bool {
+        pg_query::parse(&self.text)
+            .map(|tree| tree.select_tables() == tree.tables())


Suggested change

.map(|tree| tree.select_tables() == tree.tables())

.map(|tree| !tree.select_tables().is_empty())

Is it really necessary to check the equality? It should be enough if tree.select_tables() isn't empty, right?

mdashti · 2023-10-14T00:02:27Z

src/stats/query_result_stats.rs

+        QueryResultStats {
+            statistics: LruCache::new(NonZeroUsize::new(1000).unwrap()),
+        }


To avoid code duplication, I'd suggest:

Suggested change

QueryResultStats {

statistics: LruCache::new(NonZeroUsize::new(1000).unwrap()),

}

QueryResultStats::new()

mdashti · 2023-10-14T00:03:04Z

src/stats/query_result_stats.rs

+impl QueryResultStats {
+    pub(crate) fn new() -> QueryResultStats {
+        QueryResultStats {
+            statistics: LruCache::new(NonZeroUsize::new(1000).unwrap()),


Is it worth making this 1000 configurable? At least, it might be worth creating a named const for it.

yes, I had initially made a config option but reverted it to simplify it locally

debug log query fingerprints

2a089f1

levkk added the wip Work in progress label Jul 26, 2023

dat2 mentioned this pull request Jul 26, 2023

Implement a query blocker #61

Open

dat2 and others added 7 commits August 3, 2023 15:20

create it under the pool object

e984b87

implement query fingerprint filter

9c80348

start implementing writing to the cache

349d53a

add some tests to query parser

1cb1fa0

small things

0b7cf5c

make it work

f80c2a1

Merge branch 'main' into nickdujay/query-cache

3adfbdb

dat2 changed the title ~~[work in progress] query cache~~ [work in progress] reporting cacheable queries Aug 10, 2023

dat2 added 3 commits August 10, 2023 16:49

switch direction to a query cache statistics reporter

3ab592d

self review

7b95a06

simplify

d3ec925

dat2 changed the title ~~[work in progress] reporting cacheable queries~~ [work in progress] reporting query result statistics Aug 11, 2023

dat2 added 6 commits August 10, 2023 23:34

reorg again

d5a92bf

more info

f78247a

move sampling out of stats

8db7ecb

enable query parser

872f55f

improve query parser; implement cache eviction; update toml

721c63a

checkpoint

1d958be

jmeagher reviewed Aug 11, 2023

View reviewed changes

src/stats/query_result_stats.rs Outdated Show resolved Hide resolved

src/stats/query_result_stats.rs Outdated Show resolved Hide resolved

zainkabani reviewed Aug 14, 2023

View reviewed changes

dat2 added 5 commits August 15, 2023 16:19

turn query into a newtype

5903344

simplify PR a lot for now

afe08cc

fix tests

dcbe8bf

use pg_query parser

844ec03

use lru cache

8aa4a2c

levkk reviewed Aug 21, 2023

View reviewed changes

levkk mentioned this pull request Oct 13, 2023

Switch query parser to pg_query.rs #620

Open

mdashti reviewed Oct 14, 2023

View reviewed changes

dat2 closed this Jun 14, 2024

dat2 deleted the nickdujay/query-cache branch June 14, 2024 16:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[work in progress] reporting query result statistics #525

[work in progress] reporting query result statistics #525

Uh oh!

dat2 commented Jul 25, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

zainkabani Aug 14, 2023

Uh oh!

levkk Aug 21, 2023

Uh oh!

mdashti Oct 13, 2023

Uh oh!

dat2 Oct 20, 2023

Uh oh!

levkk Oct 20, 2023 •

edited

Loading

Uh oh!

mdashti left a comment

Uh oh!

mdashti Oct 13, 2023

Uh oh!

mdashti Oct 13, 2023

Uh oh!

dat2 Oct 19, 2023

Uh oh!

mdashti Oct 13, 2023

Uh oh!

mdashti Oct 13, 2023

Uh oh!

mdashti Oct 14, 2023

Uh oh!

mdashti Oct 14, 2023

Uh oh!

dat2 Oct 19, 2023

Uh oh!

Uh oh!

	// TODO assert length == 256
	debug_assert_eq!(hash.len(), 256)

	.map(\|tree\| tree.select_tables() == tree.tables())
	.map(\|tree\| !tree.select_tables().is_empty())

[work in progress] reporting query result statistics #525

[work in progress] reporting query result statistics #525

Uh oh!

Conversation

dat2 commented Jul 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

levkk Oct 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mdashti left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dat2 commented Jul 25, 2023 •

edited

Loading

levkk Oct 20, 2023 •

edited

Loading