docs: refine AggregateUDFImpl::is_ordered_set_aggregate documentation

[Jefffrey]

Going through some tickets related to ordered set aggregates and got a little confused on DataFusion's support for them.

As I understand it, #13511 made WITHIN GROUP mandatory for ordered set aggregate functions, of which we support only two so far:

• approx_percentile_cont
  • Technically approx_median shares some internals with approx_percentile_cont but itself isn't an ordered set aggregation
• approx_percentile_cont_with_weight (which uses approx_percentile_cont internally)

This was then amended in #16999 to make it optional, at least via the SQL API; it is still mandatory on the DataFrame API:

datafusion/datafusion/functions-aggregate/src/approx_percentile_cont.rs

Lines 53 to 58 in bbb5cc7

	/// Computes the approximate percentile continuous of a set of numbers
	pub fn approx_percentile_cont(
	order_by: Sort,
	percentile: Expr,
	centroids: Option<Expr>,
	) -> Expr {

I'm updating the doc here to try clarify things to my understanding, as a followup to the original doc update: #17744

[Jefffrey]

A question I have is if we should loosen the DataFrame API to allow omitting the sort, as #16999 did for the SQL API?

cc @alamb

[Jefffrey]

Technically we don't enforce the default order; our only ordered set aggregate functions internally use ascending as the default, so I'm not sure if we should instead say its implementation dependent or try to enforce it somehow?

[Jefffrey]

I hope I'm correct in this; some reading I used for reference: https://paquier.xyz/postgresql-2/postgres-9-4-feature-highlight-within-group/

[alamb]

This was then amended in #16999 to make it optional, at least via the SQL API; it is still mandatory on the DataFrame API:

In my mind this is mostly for backwards compatibility reasons -- #13511 basically broke a bunch of our existing user queries, so I wanted to revert the unnecessarily strict interpretation

As I understand it, #13511 made WITHIN GROUP mandatory for ordered set aggregate functions, of which we support only two so far:

Indeed -- and both of these functions have the property that many times their argument will be the same as the ORDER BY WITHIN GROUP-- for example, computing approx_median(x) implicitly means approx_median(x ORDER BY x WITHIN GROUP)

Though allowing different arguments means you can write expressions like approx_median(first_name ORDER BY salary WITHIN GROUP) and save yourself a subquery

A question I have is if we should loosen the DataFrame API to allow omitting the sort, as #16999 did for the SQL API?

cc @alamb

I suggest we hold off unless someone explicitly asks about it, though I am not opposed to it either

[alamb]

Thank you @Jefffrey -- this seems like an improvement to me

[alamb]

this is a good clarification

If DataFusion ever supports more ordered set aggregation functions, we may want to revisit this

In addition to saying what this setting doesn't do, maybe we could also say what setting it to true does do? Specifically, it seems like it only affects the output display somehow 🤔

[Jefffrey]

In addition to saying what this setting doesn't do, maybe we could also say what setting it to true does do?

This is a good point 🤔

Let me look into the code a bit more to clarify my understanding and I'll update the doc accordingly.

[Jefffrey]

In my mind this is mostly for backwards compatibility reasons -- #13511 basically broke a bunch of our existing user queries, so I wanted to revert the unnecessarily strict interpretation

I suggest we hold off unless someone explicitly asks about it, though I am not opposed to it either

I might raise a separate issue to track keeping the SQL API & DataFrame API in parity in regards to this; especially for when we consider adding more ordered set aggregate functions, if we should enforce WITHIN GROUP for those but not existing ones.

Though allowing different arguments means you can write expressions like approx_median(first_name ORDER BY salary WITHIN GROUP) and save yourself a subquery

I'm a bit confused by this example; is this just a hypothetical or something that is feasible with ordered set aggregate functions? I thought they would expected one column/expression which is the same as the ORDER BY in the WITHIN GROUP 🤔

[alamb]

I'm a bit confused by this example; is this just a hypothetical or something that is feasible with ordered set aggregate functions? I thought they would expected one column/expression which is the same as the ORDER BY in the WITHIN GROUP

I am clearly a little confused myself. Now I am not sure if there is some example where the arguments differ 🤔

[Jefffrey]

I'm a bit confused by this example; is this just a hypothetical or something that is feasible with ordered set aggregate functions? I thought they would expected one column/expression which is the same as the ORDER BY in the WITHIN GROUP

I am clearly a little confused myself. Now I am not sure if there is some example where the arguments differ 🤔

I'll move this PR back to draft and do some more research to update the docs, since it seems we're both still confused about this 😅

[Jefffrey]

Got around to doing some research.

Expectation (aka Postgres)

WITHIN GROUP clause is required for the ordered set aggregate functions; you can't omit them so an order by must always be set. The functions themselves (e.g. percentile_cont) do the sort, see this description from the postgres git commit:

Unlike the case for normal aggregates, the sorting of input rows for an ordered-set aggregate is not done behind the scenes, but is the responsibility of the aggregate's support functions. The typical implementation approach is to keep a reference to a tuplesort object in the aggregate's state value, feed the incoming rows into that object, and then complete the sorting and read out the data in the final function. This design allows the final function to perform special operations such as injecting additional hypothetical rows into the data to be sorted.

I'm not familiar with postgres internals but it reads like it expects the aggregate functions to keep the state themselves and do their own sort; WITHIN GROUP doesn't guarantee anything about input order to the function.

They also make a point about "direct" arguments:

The aggregates we have been describing so far are "normal" aggregates. PostgreSQL also supports ordered-set aggregates, which differ from normal aggregates in two key ways. First, in addition to ordinary aggregated arguments that are evaluated once per input row, an ordered-set aggregate can have "direct" arguments that are evaluated only once per aggregation operation. Second, the syntax for the ordinary aggregated arguments specifies a sort ordering for them explicitly. An ordered-set aggregate is usually used to implement a computation that depends on a specific row ordering, for instance rank or percentile, so that the sort ordering is a required aspect of any call.

For example, the percentile value like 0.5 in percentile_cont would be a direct argument.

DuckDB

DuckDB names some functions differently (quantile_cont instead of percentile_cont) and it doesn't require the WITHIN GROUP syntax:

D select quantile_cont(col0, 1) from values (1), (2) t;
┌────────────────────────┐
│ quantile_cont(col0, 1) │
│         double         │
├────────────────────────┤
│          2.0           │
└────────────────────────┘

However you can use the WITHIN GROUP syntax along with percentile_cont name:

D select percentile_cont(1) within group (order by col0) from values (1), (2) t;
┌────────────────────────────────┐
│ quantile_cont(1 ORDER BY col0) │
│             double             │
├────────────────────────────────┤
│              2.0               │
└────────────────────────────────┘
D select quantile_cont(1) within group (order by col0) from values (1), (2) t;
Parser Error:
Unknown ordered aggregate "quantile_cont".
D select percentile_cont(col0, 1) from values (1), (2) t;
Catalog Error:
Scalar Function with name percentile_cont does not exist!
Did you mean "pi"?

LINE 1: select percentile_cont(col0, 1) from values (1), (2) t;

• Note how they don't allow quantile_cont for WITHIN GROUP, only percentile_cont which is also locked to WITHIN GROUP (it cannot be used like quantile_cont)

However DuckDB allows specifying order by for quantile_cont without WITHIN GROUP:

D select quantile_cont(1 order by col0) from values (1), (2) t;
┌────────────────────────────────┐
│ quantile_cont(1 ORDER BY col0) │
│             double             │
├────────────────────────────────┤
│              2.0               │
└────────────────────────────────┘

But it doesn't allow using both:

D select quantile_cont(1 order by col0) within group (order by col0) from values (1), (2) t;
Parser Error:
cannot use multiple ORDER BY clauses with WITHIN GROUP

LINE 1: select quantile_cont(1 order by col0) within group (order by col0) from values (1), (2) t;
                                              ^

Actual (whats currently implemented in DataFusion)

(I'm ignoring anything related to schema name or unparser, focus only on functionality of aggregate).

It seems all AggregateUDFImpl::is_ordered_set_aggregate() does is during SQL parsing it does the necessary magic to put the ORDER BY column as the argument to the aggregate function:

datafusion/datafusion/sql/src/expr/function.rs

Lines 450 to 469 in c84e3cf

	let order_by = if fm.is_ordered_set_aggregate() {
	let within_group = self.order_by_to_sort_expr(
	within_group,
	schema,
	planner_context,
	false,
	None,
	)?;
	// Add the WITHIN GROUP ordering expressions to the front of the argument list
	// So function(arg) WITHIN GROUP (ORDER BY x) becomes function(x, arg)
	if !within_group.is_empty() {
	args = within_group
	.iter()
	.map(|sort| sort.expr.clone())
	.chain(args)
	.collect::<Vec<_>>();
	}
	within_group
	} else {

• i.e. essentially rewrites percentile_cont(0.5) within group (order by col0) to percentile_cont(num, 0.5) and passes along the sort order too

This is all we currently do from what I see; we don't disallow using WITHIN GROUP on other aggregate functions (see #18109). Though our DataFrame API functions for the ordered set aggregate functions do require an explicit sort order as I mentioned before:

datafusion/datafusion/functions-aggregate/src/approx_percentile_cont.rs

Lines 55 to 60 in c84e3cf

	/// Computes the approximate percentile continuous of a set of numbers
	pub fn approx_percentile_cont(
	order_by: Sort,
	percentile: Expr,
	centroids: Option<Expr>,
	) -> Expr {

References

Postgres git commit of ordered set aggregates

• Also a nice article explaining the commit a bit

Postgres documentation:

• https://www.postgresql.org/docs/9.4/sql-expressions.html#SYNTAX-AGGREGATES
• https://www.postgresql.org/docs/9.4/functions-aggregate.html#FUNCTIONS-ORDEREDSET-TABLE

DuckDB:

• https://duckdb.org/docs/stable/sql/functions/aggregates#ordered-set-aggregate-functions

[Jefffrey]

What I propose:

• We should keep WITHIN GROUP optional, since we had users using approx_percentile_cont without it (so follow DuckDB here). For other ordered set aggregate functions we follow suit (already did so for percentile_cont here feat: Add percentile_cont aggregate function #17988)
  • If WITHIN GROUP not specified then it is implementation dependent what the default order is, though we should say it is ascending (we can't control this yet via code, only comments)
• Make WITHIN GROUP strictly apply only to aggregate functions which return true for AggregateUDFImpl::is_ordered_set_aggregate() -> WITHIN GROUP needs to be more strict #18109
• AggregateUDFImpl::is_ordered_set_aggregate() itself doesn't apply any guarantees to input order, like postgres; we won't insert a sort or anything, it'll be up to the aggregate function itself to take the "hint" from WITHIN GROUP (not really a hint but you get the point) and do its ordering internally accordingly
• Refactor DataFrame APIs a bit to make it easier to use, for example:

// Currently
pub fn approx_percentile_cont(
    order_by: Sort,
    percentile: Expr,
    centroids: Option<Expr>,
) -> Expr {
    todo!()
}

// Proposed: much more explicit instead of needing to extract `expression` from `Sort`
pub fn approx_percentile_cont(
    expression: Expr,
    percentile: Expr,
    ascending: bool,
    centroids: Option<Expr>,
) -> Expr {
    todo!()
}

Thoughts @alamb ?