-
Notifications
You must be signed in to change notification settings - Fork 9.1k
HADOOP-19208: [ABFS] Fixing logic to determine HNS nature of account to avoid extra getAcl() calls #6893
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
🎊 +1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @anujmodi2021 for the patch. Added a few comments, please address it.
...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
Outdated
Show resolved
Hide resolved
...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
Outdated
Show resolved
Hide resolved
...tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestGetNameSpaceEnabled.java
Outdated
Show resolved
Hide resolved
...tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestGetNameSpaceEnabled.java
Show resolved
Hide resolved
...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
Outdated
Show resolved
Hide resolved
@rakeshadr @saxenapranav |
🎊 +1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for taking the suggestions. Looks good to me!
...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
Show resolved
Hide resolved
🎊 +1 overall
This message was automatically generated. |
Thanks @anujmodi2021 for the patch. Thanks @saxenapranav for the reviews! +1 LGTM. Pushing it to trunk shortly. |
…to avoid extra getAcl() calls (apache#6893)
…to avoid extra getAcl() calls (apache#6893)
…to avoid extra getAcl() calls (apache#6893)
Jira: https://issues.apache.org/jira/browse/HADOOP-19208
Description of PR
ABFS driver needs to know the type of account being used. It relies on the user to inform the account type using the config
fs.azure.account.hns.enabled
. If not configured, driver makes a GetAcl Head Call call to determine the account type.Expectation is getAcl() will fail with 400 Bad Request if made on the FNS Account.
If succeeded or fails with any other error code, it means it's an HNS Account.
Current Implementation:
Today when someone tries to create a new file system, getAcl is called to determine HNS/FNS account. This call fails with 404 as file system is not yet created. Then another getAcl is needed to determine HNS/FNS post filesystem creation. Leading to redundant getAcl calls.
New Implementation
This PR also fixes a test case failing on trunk.
testGetAclCallOnHnsConfigAbsence(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemInitAndCreate)
How was this patch tested?
New test around the production code changes, and existing test suite ran to validate the patch.
Test Suite Results
Metric related tests are fixed in the the #6847
:::: AGGREGATED TEST RESULT ::::
============================================================
HNS-OAuth
[ERROR] testBackoffRetryMetrics(org.apache.hadoop.fs.azurebfs.services.TestAbfsRestOperation) Time elapsed: 3.719 s <<< ERROR!
[ERROR] testReadFooterMetrics(org.apache.hadoop.fs.azurebfs.ITestAbfsReadFooterMetrics) Time elapsed: 1.167 s <<< ERROR!
[ERROR] testMetricWithIdlePeriod(org.apache.hadoop.fs.azurebfs.ITestAbfsReadFooterMetrics) Time elapsed: 1.165 s <<< ERROR!
[ERROR] testReadFooterMetricsWithParquetAndNonParquet(org.apache.hadoop.fs.azurebfs.ITestAbfsReadFooterMetrics) Time elapsed: 1.159 s <<< ERROR!
[ERROR] Tests run: 142, Failures: 0, Errors: 1, Skipped: 2
[ERROR] Tests run: 626, Failures: 0, Errors: 3, Skipped: 76
[WARNING] Tests run: 412, Failures: 0, Errors: 0, Skipped: 57
============================================================
HNS-SharedKey
[ERROR] testBackoffRetryMetrics(org.apache.hadoop.fs.azurebfs.services.TestAbfsRestOperation) Time elapsed: 3.166 s <<< ERROR!
[ERROR] testReadFooterMetrics(org.apache.hadoop.fs.azurebfs.ITestAbfsReadFooterMetrics) Time elapsed: 0.946 s <<< ERROR!
[ERROR] testMetricWithIdlePeriod(org.apache.hadoop.fs.azurebfs.ITestAbfsReadFooterMetrics) Time elapsed: 0.891 s <<< ERROR!
[ERROR] testReadFooterMetricsWithParquetAndNonParquet(org.apache.hadoop.fs.azurebfs.ITestAbfsReadFooterMetrics) Time elapsed: 0.914 s <<< ERROR!
[ERROR] Tests run: 142, Failures: 0, Errors: 1, Skipped: 3
[ERROR] Tests run: 626, Failures: 0, Errors: 3, Skipped: 28
[WARNING] Tests run: 412, Failures: 0, Errors: 0, Skipped: 44
============================================================
NonHNS-SharedKey
[ERROR] testBackoffRetryMetrics(org.apache.hadoop.fs.azurebfs.services.TestAbfsRestOperation) Time elapsed: 3.521 s <<< ERROR!
[ERROR] testReadFooterMetrics(org.apache.hadoop.fs.azurebfs.ITestAbfsReadFooterMetrics) Time elapsed: 1.003 s <<< ERROR!
[ERROR] testMetricWithIdlePeriod(org.apache.hadoop.fs.azurebfs.ITestAbfsReadFooterMetrics) Time elapsed: 0.997 s <<< ERROR!
[ERROR] testReadFooterMetricsWithParquetAndNonParquet(org.apache.hadoop.fs.azurebfs.ITestAbfsReadFooterMetrics) Time elapsed: 0.954 s <<< ERROR!
[ERROR] Tests run: 142, Failures: 0, Errors: 1, Skipped: 9
[ERROR] Tests run: 610, Failures: 0, Errors: 3, Skipped: 268
[WARNING] Tests run: 412, Failures: 0, Errors: 0, Skipped: 47
============================================================
AppendBlob-HNS-OAuth
[ERROR] testBackoffRetryMetrics(org.apache.hadoop.fs.azurebfs.services.TestAbfsRestOperation) Time elapsed: 3.712 s <<< ERROR!
[ERROR] testReadFooterMetrics(org.apache.hadoop.fs.azurebfs.ITestAbfsReadFooterMetrics) Time elapsed: 1.147 s <<< ERROR!
[ERROR] testMetricWithIdlePeriod(org.apache.hadoop.fs.azurebfs.ITestAbfsReadFooterMetrics) Time elapsed: 1.146 s <<< ERROR!
[ERROR] testReadFooterMetricsWithParquetAndNonParquet(org.apache.hadoop.fs.azurebfs.ITestAbfsReadFooterMetrics) Time elapsed: 1.199 s <<< ERROR!
[ERROR] Tests run: 142, Failures: 0, Errors: 1, Skipped: 2
[ERROR] Tests run: 626, Failures: 0, Errors: 3, Skipped: 78
[WARNING] Tests run: 412, Failures: 0, Errors: 0, Skipped: 81
Time taken: 54 mins 49 secs.