[SPARK] Support azure reads using vended credentials #416
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cloud-fan
reviewed
Sep 6, 2024
| "fs.azure.account.auth.type" -> "SAS", | ||
| "fs.azure.account.hns.enabled" -> "true", | ||
| "fs.azure.sas.token.provider.type" -> "io.unitycatalog.connectors.spark.AbfsVendedTokenProvider", | ||
| "fs.azure.sas.fixed.token" -> azCredentials.getSasToken, |
There was a problem hiding this comment.
shall we put these string constants in AbfsVendedTokenProvider?
There was a problem hiding this comment.
Good call! I cleaned that up, and pulled in the constants from abfs in cases where we were using already established conf values.
cloud-fan
reviewed
Sep 6, 2024
|
|
||
| import static java.lang.String.format; | ||
|
|
||
| public class AbfsVendedTokenProvider implements SASTokenProvider { |
There was a problem hiding this comment.
not related to this PR, do we have document to tell uses how to set this provider in Spark configs?
There was a problem hiding this comment.
Not yet -- @dennyglee and I have been working on docs for this. I'm planning on getting that polished up once we get changes in and confirm everything is working as expected
cloud-fan
approved these changes
Sep 6, 2024
vksx
pushed a commit
to vksx/unitycatalog
that referenced
this pull request
Oct 7, 2024
**PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [x] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Updates the spark connector to support vended credentials for azure cloud storage (`abfs` / `abfss`). Requires `hadoop-azure`, e.g.: ``` spark.jars.packages=org.apache.hadoop:hadoop-azure:3.3.6 ``` *Note*: this currently doesn't work with hadoop-azure:3.4.0, this issue might be addressed in 3.4.1:https://issues.apache.org/jira/browse/HADOOP-19208 Signed-off-by: Vikas Sharma <[email protected]>
kevinzwang
pushed a commit
to kevinzwang/unitycatalog
that referenced
this pull request
Oct 10, 2024
**PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [x] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Updates the spark connector to support vended credentials for azure cloud storage (`abfs` / `abfss`). Requires `hadoop-azure`, e.g.: ``` spark.jars.packages=org.apache.hadoop:hadoop-azure:3.3.6 ``` *Note*: this currently doesn't work with hadoop-azure:3.4.0, this issue might be addressed in 3.4.1:https://issues.apache.org/jira/browse/HADOOP-19208 Signed-off-by: Kevin Wang <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Checklist
docsis updatedDescription of changes
Updates the spark connector to support vended credentials for azure cloud storage (
abfs/abfss).Requires
hadoop-azure, e.g.:Note: this currently doesn't work with hadoop-azure:3.4.0, this issue might be addressed in 3.4.1:https://issues.apache.org/jira/browse/HADOOP-19208