Potential problem in uploading data using AsyncRequestBody.fromPublisher()

[masokan]

Describe the bug

Exception thrown when normal completion is expected

Expected Behavior

Upload should be successful without any exception

Current Behavior

An exception is thrown

Please see the github repository https://github.com/masokan/test-s3-upload for an example and error log

Steps to Reproduce

Please see https://github.com/masokan/test-s3-upload#readme for more details

Possible Solution

Context

I am trying to upload a stream of data for which the size is unknown

Your Environment

• AWS Java SDK version used: 2.16.68
• JDK version used: Open JDK 1.8
• Operating System and version: Linux (Ubuntu 20.04)

[debora-ito]

Hi @masokan,

software.amazon.awssdk.services.s3.model.S3Exception: You must provide the Content-Length HTTP header is a message returned by S3 saying that content-length is required for the operation you're calling. You must provide the content-length to the putObject request.

[masokan]

Hi @debora-ito, my application generates content dynamically for which the length is not known beforehand. My understanding is that AsyncRequestBody.fromPublisher() API is meant to send such content. The content is published as a series of ByteBuffer objects that the API accepts and uploads. The content length is defined to be Optional.empty() in AsyncRequestBody.java (which is in aws-java-sdk-v2.)

If you think I am using the wrong API, please let me know any other asynchronous API that I can use to upload content for which the length is unknown. Thanks.

[masokan]

Please take a look at the source code: https://github.com/masokan/test-s3-upload/blob/main/src/main/java/com/mycompany/test/StreamUpload.java
It uses AWS SDK API that does not require any content size to be specified but while running it an exception is thrown indicating content length is missing in HTTP header.

[masokan]

Hi @debora-ito I think the feature I am looking for (uploading streamed content for which length is unknown) is not supported by AWS Java SDK v2. A quick search landed me on aws/aws-sdk-java#474 where several users were looking for the same missing feature. Also, the strange thing is that the exception is thrown after all data to be uploaded is published by the Publisher. It is not thrown in the call to S3AsyncClient.putObject() itself.
I have worked with other cloud vendor libraries and they all provide streaming APIs which do not require content length to be specified upfront.
This issue is a blocker for a current project that I am working on. Please change the status as a 'feature request.' Thanks.

[debora-ito]

@masokan we are tracking this as one the many features of the V2 version of TransferManager, aws/aws-sdk-java#474 was brought out in the comments - #37 (comment).

I'll go ahead and close this, you can monitor the progress on the TransferManager v2 tracking issue - #37. Feel free to reach out with other questions.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.