Skip to content

Conversation

Jefffrey
Copy link
Contributor

Going through some tickets related to ordered set aggregates and got a little confused on DataFusion's support for them.

As I understand it, #13511 made WITHIN GROUP mandatory for ordered set aggregate functions, of which we support only two so far:

  • approx_percentile_cont
    • Technically approx_median shares some internals with approx_percentile_cont but itself isn't an ordered set aggregation
  • approx_percentile_cont_with_weight (which uses approx_percentile_cont internally)

This was then amended in #16999 to make it optional, at least via the SQL API; it is still mandatory on the DataFrame API:

/// Computes the approximate percentile continuous of a set of numbers
pub fn approx_percentile_cont(
order_by: Sort,
percentile: Expr,
centroids: Option<Expr>,
) -> Expr {

I'm updating the doc here to try clarify things to my understanding, as a followup to the original doc update: #17744

@github-actions github-actions bot added the logical-expr Logical plan and expressions label Sep 27, 2025
Copy link
Contributor Author

@Jefffrey Jefffrey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A question I have is if we should loosen the DataFrame API to allow omitting the sort, as #16999 did for the SQL API?

cc @alamb

/// calculation performed by these functions is dependent on the specific
/// sequence of the input rows, unlike other aggregate functions like `SUM`
/// `AVG`, or `COUNT`. If explit order is specified then a default order
/// of ascending is assumed.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically we don't enforce the default order; our only ordered set aggregate functions internally use ascending as the default, so I'm not sure if we should instead say its implementation dependent or try to enforce it somehow?

Comment on lines +751 to +754
/// Note that setting this to `true` does not guarantee input sort order to
/// the aggregate function; it instead gives the function full control over
/// the sorting process and they are expected to handle order of input values
/// themselves.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hope I'm correct in this; some reading I used for reference: https://paquier.xyz/postgresql-2/postgres-9-4-feature-highlight-within-group/

@Jefffrey Jefffrey marked this pull request as ready for review September 27, 2025 02:36
@alamb
Copy link
Contributor

alamb commented Sep 29, 2025

This was then amended in #16999 to make it optional, at least via the SQL API; it is still mandatory on the DataFrame API:

In my mind this is mostly for backwards compatibility reasons -- #13511 basically broke a bunch of our existing user queries, so I wanted to revert the unnecessarily strict interpretation

As I understand it, #13511 made WITHIN GROUP mandatory for ordered set aggregate functions, of which we support only two so far:

Indeed -- and both of these functions have the property that many times their argument will be the same as the ORDER BY WITHIN GROUP-- for example, computing approx_median(x) implicitly means approx_median(x ORDER BY x WITHIN GROUP)

Though allowing different arguments means you can write expressions like approx_median(first_name ORDER BY salary WITHIN GROUP) and save yourself a subquery

A question I have is if we should loosen the DataFrame API to allow omitting the sort, as #16999 did for the SQL API?

cc @alamb

I suggest we hold off unless someone explicitly asks about it, though I am not opposed to it either

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @Jefffrey -- this seems like an improvement to me

/// An example of an ordered-set aggregate function is `percentile_cont`
/// which computes a specific percentile value from a sorted list of values, and
/// is only meaningful when the input data is ordered.
/// Note that setting this to `true` does not guarantee input sort order to
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a good clarification

If DataFusion ever supports more ordered set aggregation functions, we may want to revisit this

In addition to saying what this setting doesn't do, maybe we could also say what setting it to true does do? Specifically, it seems like it only affects the output display somehow 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to saying what this setting doesn't do, maybe we could also say what setting it to true does do?

This is a good point 🤔

Let me look into the code a bit more to clarify my understanding and I'll update the doc accordingly.

@Jefffrey
Copy link
Contributor Author

In my mind this is mostly for backwards compatibility reasons -- #13511 basically broke a bunch of our existing user queries, so I wanted to revert the unnecessarily strict interpretation

I suggest we hold off unless someone explicitly asks about it, though I am not opposed to it either

I might raise a separate issue to track keeping the SQL API & DataFrame API in parity in regards to this; especially for when we consider adding more ordered set aggregate functions, if we should enforce WITHIN GROUP for those but not existing ones.

Though allowing different arguments means you can write expressions like approx_median(first_name ORDER BY salary WITHIN GROUP) and save yourself a subquery

I'm a bit confused by this example; is this just a hypothetical or something that is feasible with ordered set aggregate functions? I thought they would expected one column/expression which is the same as the ORDER BY in the WITHIN GROUP 🤔

@alamb
Copy link
Contributor

alamb commented Sep 30, 2025

I'm a bit confused by this example; is this just a hypothetical or something that is feasible with ordered set aggregate functions? I thought they would expected one column/expression which is the same as the ORDER BY in the WITHIN GROUP

I am clearly a little confused myself. Now I am not sure if there is some example where the arguments differ 🤔

@Jefffrey
Copy link
Contributor Author

Jefffrey commented Oct 1, 2025

I'm a bit confused by this example; is this just a hypothetical or something that is feasible with ordered set aggregate functions? I thought they would expected one column/expression which is the same as the ORDER BY in the WITHIN GROUP

I am clearly a little confused myself. Now I am not sure if there is some example where the arguments differ 🤔

I'll move this PR back to draft and do some more research to update the docs, since it seems we're both still confused about this 😅

@Jefffrey Jefffrey marked this pull request as draft October 1, 2025 04:37
@Jefffrey
Copy link
Contributor Author

Got around to doing some research.

Expectation (aka Postgres)

WITHIN GROUP clause is required for the ordered set aggregate functions; you can't omit them so an order by must always be set. The functions themselves (e.g. percentile_cont) do the sort, see this description from the postgres git commit:

Unlike the case for normal aggregates, the sorting of input rows for an ordered-set aggregate is not done behind the scenes, but is the responsibility of the aggregate's support functions. The typical implementation approach is to keep a reference to a tuplesort object in the aggregate's state value, feed the incoming rows into that object, and then complete the sorting and read out the data in the final function. This design allows the final function to perform special operations such as injecting additional hypothetical rows into the data to be sorted.

I'm not familiar with postgres internals but it reads like it expects the aggregate functions to keep the state themselves and do their own sort; WITHIN GROUP doesn't guarantee anything about input order to the function.

They also make a point about "direct" arguments:

The aggregates we have been describing so far are "normal" aggregates. PostgreSQL also supports ordered-set aggregates, which differ from normal aggregates in two key ways. First, in addition to ordinary aggregated arguments that are evaluated once per input row, an ordered-set aggregate can have "direct" arguments that are evaluated only once per aggregation operation. Second, the syntax for the ordinary aggregated arguments specifies a sort ordering for them explicitly. An ordered-set aggregate is usually used to implement a computation that depends on a specific row ordering, for instance rank or percentile, so that the sort ordering is a required aspect of any call.

For example, the percentile value like 0.5 in percentile_cont would be a direct argument.

DuckDB

DuckDB names some functions differently (quantile_cont instead of percentile_cont) and it doesn't require the WITHIN GROUP syntax:

D select quantile_cont(col0, 1) from values (1), (2) t;
┌────────────────────────┐
│ quantile_cont(col0, 1) │
│         double         │
├────────────────────────┤
│          2.0           │
└────────────────────────┘

However you can use the WITHIN GROUP syntax along with percentile_cont name:

D select percentile_cont(1) within group (order by col0) from values (1), (2) t;
┌────────────────────────────────┐
│ quantile_cont(1 ORDER BY col0) │
│             double             │
├────────────────────────────────┤
│              2.0               │
└────────────────────────────────┘
D select quantile_cont(1) within group (order by col0) from values (1), (2) t;
Parser Error:
Unknown ordered aggregate "quantile_cont".
D select percentile_cont(col0, 1) from values (1), (2) t;
Catalog Error:
Scalar Function with name percentile_cont does not exist!
Did you mean "pi"?

LINE 1: select percentile_cont(col0, 1) from values (1), (2) t;
  • Note how they don't allow quantile_cont for WITHIN GROUP, only percentile_cont which is also locked to WITHIN GROUP (it cannot be used like quantile_cont)

However DuckDB allows specifying order by for quantile_cont without WITHIN GROUP:

D select quantile_cont(1 order by col0) from values (1), (2) t;
┌────────────────────────────────┐
│ quantile_cont(1 ORDER BY col0) │
│             double             │
├────────────────────────────────┤
│              2.0               │
└────────────────────────────────┘

But it doesn't allow using both:

D select quantile_cont(1 order by col0) within group (order by col0) from values (1), (2) t;
Parser Error:
cannot use multiple ORDER BY clauses with WITHIN GROUP

LINE 1: select quantile_cont(1 order by col0) within group (order by col0) from values (1), (2) t;
                                              ^

Actual (whats currently implemented in DataFusion)

(I'm ignoring anything related to schema name or unparser, focus only on functionality of aggregate).

It seems all AggregateUDFImpl::is_ordered_set_aggregate() does is during SQL parsing it does the necessary magic to put the ORDER BY column as the argument to the aggregate function:

let order_by = if fm.is_ordered_set_aggregate() {
let within_group = self.order_by_to_sort_expr(
within_group,
schema,
planner_context,
false,
None,
)?;
// Add the WITHIN GROUP ordering expressions to the front of the argument list
// So function(arg) WITHIN GROUP (ORDER BY x) becomes function(x, arg)
if !within_group.is_empty() {
args = within_group
.iter()
.map(|sort| sort.expr.clone())
.chain(args)
.collect::<Vec<_>>();
}
within_group
} else {

  • i.e. essentially rewrites percentile_cont(0.5) within group (order by col0) to percentile_cont(num, 0.5) and passes along the sort order too

This is all we currently do from what I see; we don't disallow using WITHIN GROUP on other aggregate functions (see #18109). Though our DataFrame API functions for the ordered set aggregate functions do require an explicit sort order as I mentioned before:

/// Computes the approximate percentile continuous of a set of numbers
pub fn approx_percentile_cont(
order_by: Sort,
percentile: Expr,
centroids: Option<Expr>,
) -> Expr {

References

Postgres git commit of ordered set aggregates

Postgres documentation:

DuckDB:

@Jefffrey
Copy link
Contributor Author

What I propose:

  • We should keep WITHIN GROUP optional, since we had users using approx_percentile_cont without it (so follow DuckDB here). For other ordered set aggregate functions we follow suit (already did so for percentile_cont here feat: Add percentile_cont aggregate function #17988)
    • If WITHIN GROUP not specified then it is implementation dependent what the default order is, though we should say it is ascending (we can't control this yet via code, only comments)
  • Make WITHIN GROUP strictly apply only to aggregate functions which return true for AggregateUDFImpl::is_ordered_set_aggregate() -> WITHIN GROUP needs to be more strict #18109
  • AggregateUDFImpl::is_ordered_set_aggregate() itself doesn't apply any guarantees to input order, like postgres; we won't insert a sort or anything, it'll be up to the aggregate function itself to take the "hint" from WITHIN GROUP (not really a hint but you get the point) and do its ordering internally accordingly
  • Refactor DataFrame APIs a bit to make it easier to use, for example:
// Currently
pub fn approx_percentile_cont(
    order_by: Sort,
    percentile: Expr,
    centroids: Option<Expr>,
) -> Expr {
    todo!()
}

// Proposed: much more explicit instead of needing to extract `expression` from `Sort`
pub fn approx_percentile_cont(
    expression: Expr,
    percentile: Expr,
    ascending: bool,
    centroids: Option<Expr>,
) -> Expr {
    todo!()
}

Thoughts @alamb ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants