Skip to content

feat(iceberg): support remove expired files #52

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
May 23, 2025
Merged

feat(iceberg): support remove expired files #52

merged 7 commits into from
May 23, 2025

Conversation

Li0k
Copy link

@Li0k Li0k commented May 21, 2025 •
edited
Loading

Which issue does this PR close?

  • Closes #.

What changes are included in this PR?

Support clean expired files for RemoveSnapshotsAction

  1. Introducing the clear_expired_files parameter
  2. Refactor the apply function to support executing the clean function after commit
  3. Fix the issue that RewriteAction does not ignore Deleted Entry
  4. Support basic ReachableFileCleanStrategy

Are these changes tested?

@Li0k Li0k changed the title WIP: feat(iceberg): support remove expired files feat(iceberg): support remove expired files May 22, 2025
@Li0k Li0k requested review from chenzl25, xxhZs and xxchan May 22, 2025 10:36
@Li0k Li0k marked this pull request as ready for review May 22, 2025 10:39
@Li0k
Copy link
Author

Li0k commented May 23, 2025

The implementation of this PR is not perfect, we still need to implement two functions

  1. Concurrent execution of load manifest

  2. Configurable parallelism of load and delete

FYI @chenzl25 @xxhZs

chenzl25
chenzl25 approved these changes May 23, 2025
View reviewed changes
Copy link
Collaborator

@chenzl25 chenzl25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM！

@Li0k Li0k merged commit cf2aeca into dev_rebase_main_20250325 May 23, 2025
37 of 41 checks passed
@Li0k Li0k deleted the li0k/support_remove_expired_files branch May 23, 2025 09:21
@xxchan xxchan mentioned this pull request Jun 25, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chenzl25 chenzl25 chenzl25 approved these changes

@xxhZs xxhZs Awaiting requested review from xxhZs

@xxchan xxchan Awaiting requested review from xxchan

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Li0k @chenzl25