Spec additions for encryption #12162
Conversation
github-actions
bot
added
the
Specification
|
Note we'll want to start a dev list email thread once we get this nailed now to vote on the change |
|
thanks @rdblue and @RussellSpitzer for the productive call and the suggestions. I've updated the PR to reflect the discussion results. All comments and corrections are welcome. |
|
@rdblue @RussellSpitzer I've implemented the spec changes in an e2e code, everything works ok. This PR is ready for a new review round. |
|
Friendly ping on this, @rdblue / @RussellSpitzer 🙏 |
format/spec.md
Outdated
| @@ -889,6 +890,7 @@ Table metadata consists of the following fields: | |||
| | _optional_ | _optional_ | _optional_ | **`partition-statistics`** | A list (optional) of [partition statistics](#partition-statistics). | | |||
| | | | _optional_ | **`row-lineage`** | A boolean, defaulting to false, setting whether or not to track the creation and updates to rows in the table. See [Row Lineage](#row-lineage). | | |||
| | | | _optional_ | **`next-row-id`** | A `long` higher than all assigned row IDs; the next snapshot's `first-row-id`. See [Row Lineage](#row-lineage). | | |||
| | | | _optional_ | **`current-key-id`** | ID of the encryption key that encrypts the manifest list key metadata. | | |||
There was a problem hiding this comment.
I'm not sure I agree with using "current" here. I don't see how this key ID would be "current" -- it is either the key for the manifest list or it is not. How about using just key-id?
There was a problem hiding this comment.
sounds good, changing to key-id
There was a problem hiding this comment.
Sorry, my comment was based on thinking this was a snapshot's key ID because of the description. This was actually in table metadata.
Can we trust table metadata to provide a key ID to use? I would probably have the encryption manager select the key automatically -- the one that is available from KMS.
There was a problem hiding this comment.
Agreed, we can have this handled by the encryption manager.
Spec: Simplify changes needed for v3
| @@ -742,6 +743,7 @@ A snapshot consists of the following fields: | |||
| | _optional_ | _required_ | _required_ | **`summary`** | A string map that summarizes the snapshot changes, including `operation` as a _required_ field (see below) | | |||
| | _optional_ | _optional_ | _optional_ | **`schema-id`** | ID of the table's current schema when the snapshot was created | | |||
| | | | _required_ | **`first-row-id`** | The first `_row_id` assigned to the first row in the first data file in the first manifest, see [Row Lineage](#row-lineage) | | |||
| | | | _optional_ | **`key-id`** | ID of the encryption key that encrypts the manifest list key metadata | | |||
There was a problem hiding this comment.
would it make sense to name this encryption-key-id to make it more obvious that this is about encryption
There was a problem hiding this comment.
I'm not sure what other key it would be. I think it's pretty clear.
|
The vote passed, so I'll merge this. Thanks, everyone! |
No description provided.