HADOOP-13278. S3AFileSystem mkdirs does not need to validate parent path components #100
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Given S3 semantics, there's actually no problem with having a/b/c be a prefix even if a/b or a is already a file. So there's no need to check for it - it wastes API calls and can lead to problems with access control if the caller only has permissions starting at some prefix.
shanthoosh
pushed a commit
to shanthoosh/hadoop
that referenced
this pull request
Oct 15, 2019
In the processor graph, the topological sort missed adding to the visited set during graph traversal. This caused wrong graph being generated for single-node loop. This is fixed in the patch. Also fixed the maxPartition method not handling empty collection correctly. Added a few new unit tests for these. Also adjust the timing of previous async commit unit tests so it can run more reliably. Long term wise we need to fix the timer inside the AsyncRunLoop tests. Author: Xinyu Liu <[email protected]> Reviewers: Jacob Maes <[email protected]> Closes apache#100 from xinyuiscool/SAMZA-1172
singer-bin
pushed a commit
to singer-bin/hadoop
that referenced
this pull request
Dec 19, 2024
Also, cleaning up PRs: closes apache#206, closes apache#47, closes apache#72, closes apache#87, closes apache#96, closes apache#100, closes apache#107, and closes apache#112.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
According to S3 semantics, there is no conflict if a bucket contains a key named
a/band also a directory nameda/b/c. "Directories" in S3 are, after all, nothing but prefixes.However, the
mkdirscall inS3AFileSystemdoes go out of its way to traverse every parent path component for the directory it's trying to create, making sure there's no file with that name. This is suboptimal for three main reasons:mkdirs, even on a prefix that you have access to, the traversal up the path will cause you to eventually hit the root bucket, which will fail with a 403 - even though the directory creation call would have succeeded.I've opened a ticket on the Hadoop JIRA. This pull request is a simple patch that just removes this portion of the check. I have tested it with my team's instance of Spark + Luigi, and can confirm it works, and resolves the aforementioned permissions issue for a bucket on which we only had prefix access.
This is my first ticket/pull request against Hadoop, so let me know if I'm not following some convention properly :)