DOC-734 | AQL optimization: COLLECT ... AGGREGATE can utilize persistent index

[Simran-B]

Description

• index collect aggregations arangodb#21617

TODO:

• Is the performance benefit higher the fewer distinct values there are?
• Is the optimization skipped if there too many different values (low selectivity)?
• Other limitations, like sparse indexes or pre/post sort?

Upstream PRs

• 3.10:
• 3.11:
• 3.12:
• 3.13:

[arangodb-docs-automation]

Deploy Preview Available Via
https://deploy-preview-732--docs-hugo.netlify.app

[jvolmer]

To your questions:

Is the performance benefit higher the fewer distinct values there are?

We saw performance gains of around two for different numbers of values (n) and low number of distinct values (k). This gain stays constant for a lot of k values, decreases when k comes close to n and is zero when k=n. In the cluster the gain decrease happens already for lower k than in the single server case. (this behaviour can be seen in the diagrams in arangodb/arangodb#21617)

Is the optimization skipped if there too many different values (low selectivity)?

No, this optimization is not skipped based on the selectivity value - opposed to the usage of use-index-for-collect for a collect without an aggregation.

Other limitations, like sparse indexes or pre/post sort?

Yes, the optimization does not support sparse indexes, aggregation expressions with variables different than the document variable and aggregation expressions with no in-variable. I'm not sure what you mean with pre/post sort.

[jvolmer]

Suggested change

	Reading the data from the index instead of the stored documents for aggregations
	can significantly increase the perform if the there are few different values.
	Reading the data from the index instead of the stored documents for aggregations
	can increase the performance by a factor of two.