[SPARK-53272][SQL] Refactor SPJ pushdown logic out of BatchScanExec #51979
Conversation
| } | ||
| case None => | ||
| spjParams.joinKeyPositions match { | ||
| case Some(projectionPositions) => basePartitioning.partitionValues.map{r => |
There was a problem hiding this comment.
| case Some(projectionPositions) => basePartitioning.partitionValues.map{r => | |
| case Some(projectionPositions) => basePartitioning.partitionValues.map { r => |
| */ | ||
| def getInputPartitionGrouping( | ||
| p: KeyGroupedPartitioning, | ||
| spjParams: StoragePartitionJoinParams, |
There was a problem hiding this comment.
just for my curiosity: what's the relationship between p.expressions and spjParams.keyGroupedPartitioning?
There was a problem hiding this comment.
p.expressions includes join key reordering of the expressions (ref), while spjParams.keyGroupedPartitioning contains the partitioning expressions in their original ordering (which is why they must be reordered here if join key positions are present).
| */ | ||
| def getOutputKeyGroupedPartitioning( | ||
| basePartitioning: KeyGroupedPartitioning, | ||
| spjParams: StoragePartitionJoinParams): KeyGroupedPartitioning = { |
There was a problem hiding this comment.
can we move StoragePartitionJoinParams to an individual file instead of BatchScanExec.scala?
|
Also cc: @sunchao and @szehon-ho for visibility |
|
Let's file a JIRA and add it to the PR title. |
chirag-s-db
changed the title
|
@chirag-s-db can you follow the instructions and set up your Github Action? https://github.com/apache/spark/pull/51979/checks?check_run_id=47918325385 |
chirag-s-db
force-pushed
the
refactor-batch-scan
branch
from
23d337b to
1f1c551
Compare
@cloud-fan Checks should be running now, had to rebase on latest master. |
What changes were proposed in this pull request?
SPJ logic is currently closely coupled with the DSV2-specific BatchScanExec physical node, making it difficult for connectors to take advantage of SPJ for other types of scans. This PR refactors the SPJ-specific logic out of BatchScanExec, exposing a parameterized base class for connectors to use. This base class requires a partition value accessor (mapping from the parameterized type to an
InternalRow).Why are the changes needed?
Allow connectors to take advantage of SPJ on existing scans.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pure refactor - existing tests should suffice.
Was this patch authored or co-authored using generative AI tooling?
No.