Skip to content

Optional Schema Check for add_files #869

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sungwy opened this issue Jun 28, 2024 · 2 comments · Fixed by #907
Closed

Optional Schema Check for add_files #869

sungwy opened this issue Jun 28, 2024 · 2 comments · Fixed by #907
Assignees

Comments

@sungwy
Copy link
Collaborator

sungwy commented Jun 28, 2024

Feature Request / Improvement

Many folks have been reaching out about the usage of add_files, which is a sign that there's a market for users who would prefer not to rewrite parquet files and just want to side load them into an Iceberg table.

Although it is noted to be an expert user feature in the documentation (as is the case with the rest of migration procedures Iceberg has had) the ease of use of the feature seems to be appealing to users of all levels.

Therefore, I think it would be great to introduce optional guardrails for the API.

Introducing an optional schema check would be an easy first step.

@Fokko
Copy link
Contributor

Fokko commented Jun 30, 2024

@syun64 What do you think of enabling the schema validation, and having the ability to turn it off? Adding a file to a table might potentially brick a table if I understand the implication correctly.

@sungwy
Copy link
Collaborator Author

sungwy commented Jun 30, 2024

Yes, that was what I was thinking as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants