Skip to content

Document: add exactly-once delivery of Iceberg sink #479

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 5, 2025
Merged

Document: add exactly-once delivery of Iceberg sink #479

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
28213d3
add exactly-once
WanYixian May 28, 2025
a615676
remove implementation details and add note
WanYixian Jun 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions integrations/destinations/apache-iceberg.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,6 +257,14 @@ WITH (

Currently, RisingWave only supports Iceberg tables in format v2.

## Exactly-once delivery

RisingWave provides exactly-once delivery semantics for Iceberg sinks. This semantics guarantees that each data event is processed **once and only once**, even in the presence of failures such as retries or restarts. This level of delivery assurance is essential in scenarios where duplicate records can lead to incorrect analytics or data corruption in downstream systems.

Exactly-once delivery is achieved through a two-phase commit protocol involving a pre-commit phase and a commit phase. Iceberg’s commit operations are idempotent, which allows RisingWave to safely retry failed transactions without introducing duplicates.

By default, exactly-once semantics is disabled. To enable it for an Iceberg sink, include `is_exactly_once = 'true'` in the `WITH` clause of the sink definition. Note that enabling this option introduces additional coordination overhead due to metadata pre-commit, which may impact sink performance in high-throughput workloads.

## Examples

This section includes several examples that you can use if you want to quickly experiment with sinking data to Iceberg.
Expand Down
Loading