Skip to content

Add 'stripKeyPrefix' option to TransferManager's downloadDirectory method #1322

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

Add 'stripKeyPrefix' option to TransferManager's downloadDirectory method #1322

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fe004bd
Added new 'stripKeyPrefix' option to the S3 TransferManager's downloa…
kevin860 Sep 29, 2017
a95db26
Added in the actual stripping of the key prefix
kevin860 Sep 29, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 24 additions & 5 deletions aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/transfer/TransferManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1149,12 +1149,26 @@ public MultipleFileDownload downloadDirectory(String bucketName, String keyPrefi
}

public MultipleFileDownload downloadDirectory(String bucketName, String keyPrefix, File destinationDirectory, KeyFilter filter) {
return downloadDirectory(bucketName, keyPrefix, destinationDirectory, false, filter);
return downloadDirectory(bucketName, keyPrefix, destinationDirectory, false, filter, false);
}

public MultipleFileDownload downloadDirectory(String bucketName, String keyPrefix, File destinationDirectory,
boolean resumeOnRetry) {
return downloadDirectory(bucketName, keyPrefix, destinationDirectory, resumeOnRetry, null);
return downloadDirectory(bucketName, keyPrefix, destinationDirectory, resumeOnRetry, null, false);
}

public MultipleFileDownload downloadDirectory(String bucketName, String keyPrefix, File destinationDirectory, KeyFilter filter, boolean stripKeyPrefix) {
return downloadDirectory(bucketName, keyPrefix, destinationDirectory, false, filter, stripKeyPrefix);
}

public MultipleFileDownload downloadDirectory(String bucketName, String keyPrefix, File destinationDirectory,
boolean resumeOnRetry, boolean stripKeyPrefix) {
return downloadDirectory(bucketName, keyPrefix, destinationDirectory, resumeOnRetry, null, stripKeyPrefix);
}

public MultipleFileDownload downloadDirectory(String bucketName, String keyPrefix, File destinationDirectory,
boolean resumeOnRetry, KeyFilter filter) {
return downloadDirectory(bucketName, keyPrefix, destinationDirectory, resumeOnRetry, filter, false);
}

/**
Expand All @@ -1178,17 +1192,22 @@ public MultipleFileDownload downloadDirectory(String bucketName, String keyPrefi
* recursively.
* @param destinationDirectory
* The directory to place downloaded files. Subdirectories will
* be created as necessary.
* be created as necessary. The created directory structure
* will match that of the entire keys. See {@code stripKeyPrefix}
* to change this behavior.
* @param resumeOnRetry
* If set to true, upon an immediate retry of a failed object
* download, the <code>TransferManager</code> will resume the
* download from the current end of the file on disk.
* @param filter
* If set, applies the filter to determine which keys to include
* in the download request. (default is include all).
* @param stripKeyPrefix
* If set, strips the {@code keyPrefix} from the directory structure
* of the downloaded files.
*/
public MultipleFileDownload downloadDirectory(String bucketName, String keyPrefix, File destinationDirectory,
boolean resumeOnRetry, KeyFilter filter) {
boolean resumeOnRetry, KeyFilter filter, boolean stripKeyPrefix) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You never use this flag to strip the key prefix from the object key 😕

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops that's embarrassing! That's what I get for re-doing my changes in a more pull request worthy way...

Thanks, I pushed the actual functionality.

if ( keyPrefix == null )
keyPrefix = "";
if ( filter == null ) {
Expand Down Expand Up @@ -1268,7 +1287,7 @@ public MultipleFileDownload downloadDirectory(String bucketName, String keyPrefi

for ( S3ObjectSummary summary : objectSummaries ) {
// TODO: non-standard delimiters
File f = new File(destinationDirectory, summary.getKey());
File f = new File(destinationDirectory, stripKeyPrefix ? summary.getKey().substring(keyPrefix.length()) : summary.getKey());
File parentFile = f.getParentFile();

if ( !parentFile.exists() && !parentFile.mkdirs() ) {
Expand Down