Support S3TransferManager upload using Flux<ByteBuffer>

[pkgonan]

Describe the Feature

Support S3TransferManager upload using Flux.
Consider changing UploadRequest's source File to Flux.

Additional Context

• aws-sdk-java-v2/services-custom/s3-transfer-manager/src/main/java/software/amazon/awssdk/transfer/s3/UploadRequest.java

  Line 39 in ca8aa92

  private final Path source;

• aws-sdk-java-v2/services-custom/s3-transfer-manager/src/main/java/software/amazon/awssdk/transfer/s3/internal/DefaultS3TransferManager.java

  Line 91 in df8b220

  return AsyncRequestBody.fromFile(uploadRequest.source());

• Transfer Manager #37

[pkgonan]

@zoewangg @debora-ito
Hi. Please consider this feature request.

[debora-ito]

Thank you for reaching out @pkgonan, feature request acknowledged.

[Bennett-Lynch]

Hi @pkgonan. I'll be taking a look at this feature request. It may entail backwards incompatible API changes as we try to settle on the correct API (since TransferManager is still in PREVIEW). I'll include you on the pull request once it's ready.

[Bennett-Lynch]

This has been resolved with #2817. Thanks for the feature request, @pkgonan!

As mentioned in the PR, you should be able to adapt a Flux<ByteBuffer> to an AsyncRequestBody by doing something like: AsyncRequestBody.fromPublisher(Flux<ByteBuffer>).

On the response/download side, this is currently a little bit more complicated. You would need to implement your own AsyncResponseTransformer and capture the SdkPublisher<ByteBuffer> given as part of the onStream(..) method. If this is a common use case, I would like to consider making this more convenient by offering a standard SDK implementation to do this.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[djchapm]

@pkgonan did you ever get this to work? "AsyncRequestBody.fromPublisher(Flux)"

Still complains about content length if I try to do something like this with latest transferManager in v2.17.210-PREVIEW