Skip to content

Implement column projection #1443

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Feb 10, 2025
Merged

Conversation

gabeiglio
Copy link
Contributor

@gabeiglio gabeiglio commented Dec 18, 2024

This is a fix for issue #1401. In which table scans needed to infer partition column by following the column projection rules

Fixes #1401

@Fokko Fokko self-requested a review December 18, 2024 20:38
@gabeiglio gabeiglio marked this pull request as ready for review December 19, 2024 15:12
Copy link
Contributor

@kevinjqliu kevinjqliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a few comments, please take a look! The PR looks great already. Thanks for working on this!

@kevinjqliu kevinjqliu self-requested a review December 23, 2024 19:04
…tion logic to helper method, changed test to use high-level table scan
@Fokko Fokko self-requested a review January 13, 2025 12:48
Copy link
Contributor

@kevinjqliu kevinjqliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

generally LGTM! I added a few nit comments and some clarifying questions on testing.

thanks for working on this!

Copy link
Contributor

@kevinjqliu kevinjqliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like CI caught an interesting case where a new identity partition is added after data files were written. The accessor then cannot find the proper partition record... We need to do something like this

Copy link
Contributor

@Fokko Fokko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there!

Copy link
Contributor

@kevinjqliu kevinjqliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks @gabeiglio !!

@kevinjqliu kevinjqliu requested a review from Fokko February 5, 2025 02:16
@Fokko Fokko merged commit d9b1c03 into apache:main Feb 10, 2025
7 checks passed
@Fokko
Copy link
Contributor

Fokko commented Feb 10, 2025

Thanks for working on this @gabeiglio, and thanks for the review @kevinjqliu 🙌

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

API table.scan does not conform to Iceberg spec for identity partition columns
3 participants