-
Notifications
You must be signed in to change notification settings - Fork 1.7k
docs: refine AggregateUDFImpl::is_ordered_set_aggregate
documentation
#17805
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/// calculation performed by these functions is dependent on the specific | ||
/// sequence of the input rows, unlike other aggregate functions like `SUM` | ||
/// `AVG`, or `COUNT`. If explit order is specified then a default order | ||
/// of ascending is assumed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Technically we don't enforce the default order; our only ordered set aggregate functions internally use ascending as the default, so I'm not sure if we should instead say its implementation dependent or try to enforce it somehow?
/// Note that setting this to `true` does not guarantee input sort order to | ||
/// the aggregate function; it instead gives the function full control over | ||
/// the sorting process and they are expected to handle order of input values | ||
/// themselves. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I hope I'm correct in this; some reading I used for reference: https://paquier.xyz/postgresql-2/postgres-9-4-feature-highlight-within-group/
In my mind this is mostly for backwards compatibility reasons -- #13511 basically broke a bunch of our existing user queries, so I wanted to revert the unnecessarily strict interpretation
Indeed -- and both of these functions have the property that many times their argument will be the same as the Though allowing different arguments means you can write expressions like
I suggest we hold off unless someone explicitly asks about it, though I am not opposed to it either |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @Jefffrey -- this seems like an improvement to me
/// An example of an ordered-set aggregate function is `percentile_cont` | ||
/// which computes a specific percentile value from a sorted list of values, and | ||
/// is only meaningful when the input data is ordered. | ||
/// Note that setting this to `true` does not guarantee input sort order to |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is a good clarification
If DataFusion ever supports more ordered set aggregation functions, we may want to revisit this
In addition to saying what this setting doesn't do, maybe we could also say what setting it to true
does do? Specifically, it seems like it only affects the output display somehow 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In addition to saying what this setting doesn't do, maybe we could also say what setting it to
true
does do?
This is a good point 🤔
Let me look into the code a bit more to clarify my understanding and I'll update the doc accordingly.
I might raise a separate issue to track keeping the SQL API & DataFrame API in parity in regards to this; especially for when we consider adding more ordered set aggregate functions, if we should enforce
I'm a bit confused by this example; is this just a hypothetical or something that is feasible with ordered set aggregate functions? I thought they would expected one column/expression which is the same as the |
I am clearly a little confused myself. Now I am not sure if there is some example where the arguments differ 🤔 |
I'll move this PR back to draft and do some more research to update the docs, since it seems we're both still confused about this 😅 |
Got around to doing some research. Expectation (aka Postgres)
I'm not familiar with postgres internals but it reads like it expects the aggregate functions to keep the state themselves and do their own sort; They also make a point about "direct" arguments:
For example, the percentile value like DuckDB DuckDB names some functions differently ( D select quantile_cont(col0, 1) from values (1), (2) t;
┌────────────────────────┐
│ quantile_cont(col0, 1) │
│ double │
├────────────────────────┤
│ 2.0 │
└────────────────────────┘ However you can use the D select percentile_cont(1) within group (order by col0) from values (1), (2) t;
┌────────────────────────────────┐
│ quantile_cont(1 ORDER BY col0) │
│ double │
├────────────────────────────────┤
│ 2.0 │
└────────────────────────────────┘
D select quantile_cont(1) within group (order by col0) from values (1), (2) t;
Parser Error:
Unknown ordered aggregate "quantile_cont".
D select percentile_cont(col0, 1) from values (1), (2) t;
Catalog Error:
Scalar Function with name percentile_cont does not exist!
Did you mean "pi"?
LINE 1: select percentile_cont(col0, 1) from values (1), (2) t;
However DuckDB allows specifying order by for D select quantile_cont(1 order by col0) from values (1), (2) t;
┌────────────────────────────────┐
│ quantile_cont(1 ORDER BY col0) │
│ double │
├────────────────────────────────┤
│ 2.0 │
└────────────────────────────────┘ But it doesn't allow using both: D select quantile_cont(1 order by col0) within group (order by col0) from values (1), (2) t;
Parser Error:
cannot use multiple ORDER BY clauses with WITHIN GROUP
LINE 1: select quantile_cont(1 order by col0) within group (order by col0) from values (1), (2) t;
^ Actual (whats currently implemented in DataFusion) (I'm ignoring anything related to schema name or unparser, focus only on functionality of aggregate). It seems all datafusion/datafusion/sql/src/expr/function.rs Lines 450 to 469 in c84e3cf
This is all we currently do from what I see; we don't disallow using
References Postgres git commit of ordered set aggregates
Postgres documentation:
DuckDB: |
What I propose:
// Currently
pub fn approx_percentile_cont(
order_by: Sort,
percentile: Expr,
centroids: Option<Expr>,
) -> Expr {
todo!()
}
// Proposed: much more explicit instead of needing to extract `expression` from `Sort`
pub fn approx_percentile_cont(
expression: Expr,
percentile: Expr,
ascending: bool,
centroids: Option<Expr>,
) -> Expr {
todo!()
} Thoughts @alamb ? |
Going through some tickets related to ordered set aggregates and got a little confused on DataFusion's support for them.
As I understand it, #13511 made
WITHIN GROUP
mandatory for ordered set aggregate functions, of which we support only two so far:approx_percentile_cont
approx_median
shares some internals withapprox_percentile_cont
but itself isn't an ordered set aggregationapprox_percentile_cont_with_weight
(which usesapprox_percentile_cont
internally)This was then amended in #16999 to make it optional, at least via the SQL API; it is still mandatory on the DataFrame API:
datafusion/datafusion/functions-aggregate/src/approx_percentile_cont.rs
Lines 53 to 58 in bbb5cc7
I'm updating the doc here to try clarify things to my understanding, as a followup to the original doc update: #17744