feat: add process delete enrty in snapshot produce #33
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ZENOTME
force-pushed
the
zj/process_delete_entry_mr
branch
2 times, most recently
from
af5d27f to
a81ef9a
Compare
ZENOTME
force-pushed
the
zj/process_delete_entry_mr
branch
from
a81ef9a to
59af94f
Compare
Li0k
approved these changes
Mar 18, 2025
crates/iceberg/src/transaction.rs
Outdated
| self.new_manifest_writer(&ManifestContentType::Data, spec_id)?, | ||
| ); | ||
| } | ||
| data_file_writer.as_mut().unwrap().add_entry(entry)?; |
Li0k
requested changes
Mar 18, 2025
crates/iceberg/src/transaction.rs
Outdated
| self.new_manifest_writer(&ManifestContentType::Deletes, spec_id)?, | ||
| ); | ||
| } | ||
| delete_file_writer.as_mut().unwrap().add_entry(entry)?; |
There was a problem hiding this comment.
Pull Request Overview
This PR adds support for delete manifest processing in snapshot production by introducing a new partition_spec_id parameter. It also updates various builders and tests across the codebase to propagate this new field.
- Added partition_spec_id to the manifest writer functions and related builders.
- Implemented new delete manifest writing logic in transaction and snapshot production.
- Updated test files and file writers to include the additional partition_spec_id parameter.
Reviewed Changes
Copilot reviewed 21 out of 21 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| crates/iceberg/src/transaction.rs | Updated new_manifest_writer signature and delete manifest logic. |
| crates/iceberg/src/scan.rs crates/iceberg/src/io/object_cache.rs |
Added partition_spec_id to DataFileBuilder invocations in tests. |
| crates/iceberg/src/writer/base_writer/data_file_writer.rs | Modified builder construction to include partition_spec_id parameter. |
| crates/integration_tests/tests/shared_tests/append_partition_data_file_test.rs | Updated builder calls to supply partition_spec_id. |
| crates/iceberg/src/writer/base_writer/sort_position_delete_writer.rs | Incorporated partition_spec_id into sort position delete logic. |
| crates/iceberg/src/writer/base_writer/equality_delete_writer.rs | Revised equality delete writer config and tests for partition_spec_id. |
| crates/iceberg/src/arrow/record_batch_partition_spliter.rs | Adjusted import and added accessor for partition_spec. |
| crates/iceberg/src/spec/manifest.rs | Updated manifest entry conversion to pass partition_spec_id. |
| crates/iceberg/src/writer/function_writer/fanout_partition_writer.rs | Added partition_spec_id rewriting to data files. |
| crates/iceberg/src/writer/file_writer/parquet_writer.rs | Set partition_spec_id in file writer and updated tests accordingly. |
| (Other files have similar updates to builders and tests to pass partition_spec_id.) |
Co-authored-by: Copilot <[email protected]>
Li0k
pushed a commit
that referenced
this pull request
Mar 25, 2025
* support spec id in data file * support proccess delete entry * fullfill partition spec id * fix * fix spelling mistake Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: ZENOTME <[email protected]> Co-authored-by: Copilot <[email protected]>
xxchan
pushed a commit
that referenced
this pull request
Mar 25, 2025
* support spec id in data file * support proccess delete entry * fullfill partition spec id * fix * fix spelling mistake Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: ZENOTME <[email protected]> Co-authored-by: Copilot <[email protected]>
xxchan
pushed a commit
that referenced
this pull request
Mar 25, 2025
* support spec id in data file * support proccess delete entry * fullfill partition spec id * fix * fix spelling mistake Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: ZENOTME <[email protected]> Co-authored-by: Copilot <[email protected]>
xxchan
pushed a commit
that referenced
this pull request
Mar 25, 2025
* support spec id in data file * support proccess delete entry * fullfill partition spec id * fix * fix spelling mistake Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: ZENOTME <[email protected]> Co-authored-by: Copilot <[email protected]> Signed-off-by: xxchan <[email protected]>
xxchan
pushed a commit
that referenced
this pull request
Mar 25, 2025
* support spec id in data file * support proccess delete entry * fullfill partition spec id * fix * fix spelling mistake Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: ZENOTME <[email protected]> Co-authored-by: Copilot <[email protected]> Signed-off-by: xxchan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
What changes are included in this PR?
Are these changes tested?