Skip to content

[work in progress] reporting query result statistics #525

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 22 commits into from

Conversation

dat2
Copy link
Contributor

@dat2 dat2 commented Jul 25, 2023

in order to discover what queries are cacheable, we need to know roughly how many times we get the same result for a given query. this implements that by hashing a specific query text, hashing the results we get back, and storing some metadata such as first_seen, last_seen, and count. we can compute a rough TTL for cacheable queries using last_seen - first_seen to see how often query results change (for the exact same query text).

on our databases, we have some tables that see thousands of writes / week, but billions of reads / week, so we're trying to estimate how long we can cache those rows to try and take the load off for those billions of reads.

i'm able to make this work with

  • LOG_LEVEL=debug cargo run -- pgcat.minimal.toml
  • PGPASSWORD=postgres psql -p 6432 -h localhost -d pgml -U postgres -w -c 'select 1'
  • PGPASSWORD=postgres psql -p 6432 -h localhost -d pgml -U postgres -w -c 'select 2'
  • PGPASSWORD=postgres psql -p 6432 -h localhost -d pgml -U postgres -w -c 'select 3'
  • wait some time
  • PGPASSWORD=postgres psql -p 6432 -h localhost -d pgml -U postgres -w -c 'select 1'
  • PGPASSWORD=pgcat psql -p 6432 -h localhost -d pgcat -U pgcat -w -c 'show query_result_stats'
 database |   user   | normalized |     fingerprint     |                            query_hash                            |                           result_hash                            | count |           first_seen           |           last_seen            |    duration
----------+----------+------------+---------------------+------------------------------------------------------------------+------------------------------------------------------------------+-------+--------------------------------+--------------------------------+-----------------
 pgml     | postgres | select $1  | 5836069208177285818 | 7c10e523b62b91c18b1583477de02a827c10e523b62b91c18b1583477de02a82 | 53d70edc5bb4891237a956e17423bfe153d70edc5bb4891237a956e17423bfe1 |     5 | 2023-08-11 13:39:48.633979 UTC | 2023-08-11 13:40:08.208047 UTC | 0d 00:00:19.574
 pgml     | postgres | select $1  | 5836069208177285818 | 3c9c5d447eb1ce24bc442b78cc88f93c3c9c5d447eb1ce24bc442b78cc88f93c | bd131b4cebad41e69e9d8d4f0cda706bd131b4cebad41e69e9d8d4f0cda706   |     1 | 2023-08-11 13:40:04.734028 UTC | 2023-08-11 13:40:04.734029 UTC | 0d 00:00:00.000
 pgml     | postgres | select $1  | 5836069208177285818 | 549b8517156cb240c57f52e7f2f2fa3e549b8517156cb240c57f52e7f2f2fa3e | e14fcf3615636eecb874f0d2f6ca287ee14fcf3615636eecb874f0d2f6ca287e |     1 | 2023-08-11 13:40:05.872017 UTC | 2023-08-11 13:40:05.872018 UTC | 0d 00:00:00.000
(3 rows)

@levkk levkk added the wip Work in progress label Jul 26, 2023
@dat2 dat2 mentioned this pull request Jul 26, 2023
@dat2 dat2 changed the title [work in progress] query cache [work in progress] reporting cacheable queries Aug 10, 2023
@dat2 dat2 changed the title [work in progress] reporting cacheable queries [work in progress] reporting query result statistics Aug 11, 2023
src/client.rs Outdated
@@ -1675,6 +1678,20 @@ where
Some(message) => message,
None => &self.buffer,
};
// TODO query sample rate from config
let should_parse_query = pool.settings.query_parser_enabled
&& !server.in_transaction()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also make sure it's not a begin statement we're caching or a set statement? Ideally we just check for select from

@@ -3,6 +3,9 @@ name = "pgcat"
version = "1.1.1"
edition = "2021"

[profile.release]
debug = true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The executable is gonna be quite large. Not an issue for production imo.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really required, though?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I committed it unnecessarily

Copy link
Contributor

@levkk levkk Oct 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's great for debugging production issues, otherwise the backtraces are pretty slim. This just attaches debug symbols, but still compiles it with all the optimizations.

Copy link
Contributor

@mdashti mdashti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dat2 Thanks for the PR. This is certainly exciting.

@@ -3,6 +3,9 @@ name = "pgcat"
version = "1.1.1"
edition = "2021"

[profile.release]
debug = true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really required, though?

@@ -48,6 +51,8 @@ itertools = "0.10"
clap = { version = "4.3.1", features = ["derive", "env"] }
tracing = "0.1.37"
tracing-subscriber = { version = "0.3.17", features = ["json", "env-filter", "std"]}
pg_query = "0.7"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're adding pg_query, shouldn't we also remove sqlparser (and replace its usages with pg_query)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes! feel free to do it. I wont' be able to get to it for a month or so

@@ -40,3 +41,10 @@ pub fn format_duration(duration: &chrono::Duration) -> String {
days, hours, minutes, seconds, milliseconds
)
}

pub fn hash_string(hash: &[u8]) -> String {
// TODO assert length == 256
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// TODO assert length == 256
debug_assert_eq!(hash.len(), 256)

Is that enough?


pub fn is_select(&self) -> bool {
pg_query::parse(&self.text)
.map(|tree| tree.select_tables() == tree.tables())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
.map(|tree| tree.select_tables() == tree.tables())
.map(|tree| !tree.select_tables().is_empty())

Is it really necessary to check the equality? It should be enough if tree.select_tables() isn't empty, right?

Comment on lines +33 to +35
QueryResultStats {
statistics: LruCache::new(NonZeroUsize::new(1000).unwrap()),
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To avoid code duplication, I'd suggest:

Suggested change
QueryResultStats {
statistics: LruCache::new(NonZeroUsize::new(1000).unwrap()),
}
QueryResultStats::new()

impl QueryResultStats {
pub(crate) fn new() -> QueryResultStats {
QueryResultStats {
statistics: LruCache::new(NonZeroUsize::new(1000).unwrap()),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth making this 1000 configurable? At least, it might be worth creating a named const for it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I had initially made a config option but reverted it to simplify it locally

@dat2 dat2 closed this Jun 14, 2024
@dat2 dat2 deleted the nickdujay/query-cache branch June 14, 2024 16:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
wip Work in progress
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants