HADOOP-19681: Fix S3A failing to initialize S3 buckets having namespace with dot followed by number #7942
+95
−8
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test |
|
@steveloughran : Could you please review the changes. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
steveloughran
requested changes
Sep 15, 2025
There was a problem hiding this comment.
Oh, this complicates things. I've been happily treating all issues related to buckets with . in them as WONTFIX. Storediag tells people off too.
- make sure that FileSystem.get(URI, config) always returns the same instance from cache (and a different one from that with a a .2 as a suffix)
- could this be moved to a unit test, rather than an ITest?
- you will need to add something in the documentation, especially about path style access -and warn that support is not guaranteed across other versions/applications.
| Configuration config = new Configuration(); | ||
| Path path = new Path("s3a://test-bucket-v1.1"); | ||
| try (FileSystem fs = path.getFileSystem(config)) { | ||
| assertThat(fs instanceof S3AFileSystem) |
There was a problem hiding this comment.
use whatever assertj assertion is about instanceof, so you get a better error.
@steveloughran, in HADOOP-17241, you referenced this announcement as one of the reasons to not support this buckets with dot in the name: https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/ However, AWS have since revised their stance. AWS has confirmed they will continue supporting buckets with dots in their names through virtual hosted-style URLs due to customer feedback and compatibility requirements.
So I guess it makes sense to add support for it in S3A. |
…ce with dot followed by number
shameersss1
force-pushed
the
HADOOP-19681
branch
from
ea1308a to
5014c32
Compare
|
Thanks @steveloughran for the review. I have addressed your comments. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
S3A fails to initialize when S3 bucket namespace is having dot followed by a number.
Specific Problem: URI parsing fails when S3 bucket names contain a dot followed by a number (like bucket-v1.1-us-east-1). Java's
URI.getHost() method incorrectly interprets the dot-number pattern as a port specification, causing it to return null.
How was this patch tested?
Tested in us-east-1 with bucket having namespace with dot followed by a number.
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?