Refactor S3TransferManager to support non-file-based transfers

[Bennett-Lynch]

Revision 2 (ffc3deb, 4aac24a)

1. Update change log to indicate breaking changes
2. Update README to reflect new API and also to include examples for TransferListener
3. Misc. small cleanups from feedback
4. Additional test coverage

Revision 1 (995ca7d)

Motivation and Context

We would like for S3TransferManager to support non-file-based uploads and downloads, such as reactive streams or in-memory byte buffers, but the current implementation is directly coupled to file/Path-based transfers. While the underlying S3AsyncClient already has support for AsyncRequestBodies and AsyncResponseTransformers, these constructs are not exposed through the current S3TransferManager interface.

Additionally, the existing S3TransferManager interfaces have made some assumptions that make it difficult to introduce this support, such as UploadRequest/DownloadRequest having a Path attribute, and not having interfaces that allow us to distinguish between transfer types, e.g., single-object and multi-object (directory) transfers. Likewise, the current interface hierarchy also makes it difficult to distinguish between file-based and non-file-based. Therefore, this refactoring requires backwards-incompatible changes, which are permissible while S3TransferManager is still in PREVIEW.

Description

The diff here is very large, but this change set primarily introduces the following interface hierarchy:

---

• TransferRequest
  • TransferObjectRequest
    • UploadFileRequest
    • DownloadFileRequest
    • UploadRequest
    • DownloadRequest<T>
  • TransferDirectoryRequest
    • UploadDirectoryRequest
• Transfer
  • ObjectTransfer
    • FileUpload
    • FileDownload
    • Upload
    • Download<T>
  • DirectoryTransfer
    • DirectoryUpload
• CompletedTransfer
  • CompletedObjectTransfer
    • CompletedFileUpload
    • CompletedFileDownload
    • CompletedUpload
    • CompletedDownload<T>
  • CompletedDirectoryTransfer
    • CompletedDirectoryUpload
• FailedObjectTransfer
  • FailedFileUpload

---

These interfaces allow us to more selectively choose which attributes will be shared between different data types. Additionally, we take this opportunity to make the naming convention and class-vs-interface distinction consistent across all the above data types.

It is important that we distinguish between file-based-requests and non-file-based-requests so that the Collection<> of failed file transfers from a directory upload can be re-driven, if needed.

Note that the above Download-oriented data types are all generic. This is because of how AsyncResponseTransformer is designed to be generic, being parameterized with the type of data that a given AsyncResponseTransformer produces. Therefore, a user must declare the type at the time of instantiating a DownloadRequest, and the type will
be consistent throughout the transfer lifecycle, including the Download and CompletedDownload interfaces.

For a quick overview of the new API, refer to the upload/download integration tests:

Arbitrary object transfers

aws-sdk-java-v2/services-custom/s3-transfer-manager/src/it/java/software/amazon/awssdk/transfer/s3/S3TransferManagerUploadIntegrationTest.java

Lines 92 to 99 in 995ca7d

	Upload upload =
	tm.upload(UploadRequest.builder()
	.putObjectRequest(b -> b.bucket(TEST_BUCKET).key(TEST_KEY))
	.requestBody(AsyncRequestBody.fromString(content))
	.overrideConfiguration(b -> b.addListener(LoggingTransferListener.create()))
	.build());
	CompletedUpload completedUpload = upload.completionFuture().join();

aws-sdk-java-v2/services-custom/s3-transfer-manager/src/it/java/software/amazon/awssdk/transfer/s3/S3TransferManagerDownloadIntegrationTest.java

Lines 79 to 86 in 995ca7d

	Download<ResponseBytes<GetObjectResponse>> download =
	tm.download(DownloadRequest.builder()
	.getObjectRequest(b -> b.bucket(BUCKET).key(KEY))
	.responseTransformer(AsyncResponseTransformer.toBytes())
	.overrideConfiguration(b -> b.addListener(LoggingTransferListener.create()))
	.build());
	CompletedDownload<ResponseBytes<GetObjectResponse>> completedDownload = download.completionFuture().join();
	ResponseBytes<GetObjectResponse> result = completedDownload.result();

File-based transfers

aws-sdk-java-v2/services-custom/s3-transfer-manager/src/it/java/software/amazon/awssdk/transfer/s3/S3TransferManagerUploadIntegrationTest.java

Lines 69 to 76 in 995ca7d

	FileUpload fileUpload =
	tm.uploadFile(UploadFileRequest.builder()
	.putObjectRequest(b -> b.bucket(TEST_BUCKET).key(TEST_KEY))
	.source(testFile.toPath())
	.overrideConfiguration(b -> b.addListener(LoggingTransferListener.create()))
	.build());
	CompletedFileUpload completedFileUpload = fileUpload.completionFuture().join();

aws-sdk-java-v2/services-custom/s3-transfer-manager/src/it/java/software/amazon/awssdk/transfer/s3/S3TransferManagerDownloadIntegrationTest.java

Lines 66 to 72 in 995ca7d

	FileDownload download =
	tm.downloadFile(DownloadFileRequest.builder()
	.getObjectRequest(b -> b.bucket(BUCKET).key(KEY))
	.destination(path)
	.overrideConfiguration(b -> b.addListener(LoggingTransferListener.create()))
	.build());
	CompletedFileDownload completedFileDownload = download.completionFuture().join();

License

• I confirm that this pull request can be released under the Apache 2 license

[Bennett-Lynch]

/cc @pkgonan & #2714. Please let me know if this API satisfies your use case. You should be able to adapt a Flux<ByteBuffer> to this API by doing something like: AsyncRequestBody.fromPublisher(Flux<ByteBuffer>).

[pkgonan]

@Bennett-Lynch
Thank you so much. This is exactly the feature I've been waiting for. thank you.

[zoewangg]

Can we update the sample code in the README? cc @Carey-AWS, can we update dev guide?

[Bennett-Lynch]

Can we update the sample code in the README?

@zoewangg I updated the README. I wasn't sure if we should include examples for the arbitrary upload/download, or just file transfers (to keep it simple). Please take a look at let me know what you prefer.

[zoewangg]

Thanks for updating the README, LGTM. I think we should have sample code for common use-cases like what you added. The DDB enhanced client README is a good example, imo and we've got positive feedback on it. https://github.com/aws/aws-sdk-java-v2/tree/master/services-custom/dynamodb-enhanced

[sonarqubecloud]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
30 Code Smells

72.3% Coverage
6.7% Duplication