"java.net.SocketException: Broken pipe (Write failed): error on file upload (reproducible + workaround)

[KamilSucharski]

Describe the bug

When uploading a certain .zip file (below) to an AWS S3 bucket with the Java SDK I encountered an error "java.net.SocketException: Broken pipe (Write failed)". This error does not appear when uploading with web gui or amazon cli. It also does not appear when I put a password on the .zip file, or split the contents in two separate .zip files. Size should not be an issue here since it's only 33 megabytes.

File link:
https://drive.google.com/file/d/1DqY6NR2lKdFqpz2kO95pXYqxsvb0e9me/view?usp=sharing

If the file link dies, source is this website (click download under "Download Anonymized DICOM Study":
https://www.dicomlibrary.com/?manage=1b9baeb16d2aeba13bed71045df1bc65

Possible Solution

The workaround until this is fixed is to use ObjectMetadata with Content-Length set, even though the documentation says that users cannot modify this value. Without it the SDK will end the connection prematurely. So instead of doing this:

inputStream = multipartFile.getInputStream();
amazonS3.putObject(bucketName, key, inputStream, null);

Do this:

inputStream = multipartFile.getInputStream();
long contentLength = multipartFile.getSize();
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setContentLength(contentLength);
amazonS3.putObject(bucketName, key, inputStream, objectMetadata);

Your Environment

• AWS Java SDK version used: 1.11.860
• JDK version used: 1.8
• Operating System and version: any

Stacktrace

com.amazonaws.SdkClientException: Unable to execute HTTP request: Broken pipe (Write failed)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5155)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5101)
        at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:396)
        at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6048)
        at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1814)
        at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1774)
        at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1712)

[debora-ito]

@KamilSucharski apologies for the long pause here. If I understood the issue correctly you are trying to upload an InputStream to S3 and it doesn't work unless you provide the part's content-length. This is an S3 limitation, S3 requires the content-length in advance for the request signature. There's some issues related to the subject: #474, #1748.

Providing the content-length in the ObjectMetadata is a good workaround. Let us know if you have any other question.

[KamilSucharski]

Okay, you are right. This documentation confirms that "This field is required when uploading objects to S3, but the AWS S3 Java client will automatically set it when working directly with files. When uploading directly from a stream, set this field if possible.".

What threw me off is that it worked from the CLI and also worked for many other files. Also this page says "Can User Modify the Value: No", but I guess it's not about upload.

Thanks, closing this issue.