Support Uploading to S3 using an OutputStream

[sixcorners]

Aren't we outputting to S3? Shouldn't we be getting an output stream to resources in S3 to write stuff?
spring-cloud-aws has an implementation of this here:
SimpleStorageResource

[kiiadi]

An InputStream allows the SDK to pull data into S3. What's your use case exactly? In a future version of the SDK we're looking at changing the streaming API a bit. It would be good to understand what you're trying to do.

[sixcorners]

I guess I'm not doing anything that outputs to OutputStreams anymore so I don't have a specific need. But I still think it makes sense that APIs that let you write files should return OutputStreams.
The functionality that would actually be helpful to me would be a function that uploads streams (input or output streams) without buffering it to determine Content-Length like how that SimpleStorageResource class does it but I guess I can create a new issue for that.

[dagnir]

Going to go ahead close this issue. @sixcorners we're currently working on the next iteration of the SDK which address some of these drawbacks so please be on the lookout for it!

[martindvds]

This would be a highly appreciated feature, since it would make it a lot easier to plug S3 into places where you already write into a FileOutputStream, without having to redesign anything.

[123tharun]

Is this implemented yet?. @dagnir @kiiadi @sixcorners

We are generating documents using doc4jx which gives us an outputstream, we do not want to store in a document and upload it to the S3. We would directly like to store the inputstream from doc4jx directly to the S3.

[kvsravindrareddy]

I really appreciate if you could provide some examples to write output stream directly S3 bucket

[guerard]

@kvsravindrareddy You can only provide an InputStream to the AmazonS3 client for it to read from. You'll need to know the length in bytes of the content prior to uploading if you want it to actually be streamed. AmazonS3 will otherwise just buffer the entire contents in-memory and then send it off.

If you do know how many bytes you're sending, then you could pipe streams e.g.:

long numBytes;  // length of data to send in bytes..somehow you know it before processing the entire stream
PipedOutputStream os = new PipedOutputStream();
PipedInputStream is = new PipedInputStream(os);
ObjectMetadata meta = new ObjectMetadata();
meta.setContentLength(numBytes);
new Thread(() -> {
    /* Write to os here; make sure to close it when you're done */
}).start();
amazonS3.putObject("myBucket", "myKey", is, meta);
is.close();  // always close your streams

The question is how does one convert an OutputStream to an InputStream. Alternatives are to write to a temp File or buffer it all in-memory. If you don't know the size, then you won't see any benefit to pipes w/ AmazonS3 compared to using ByteArrayOutputStream + ByteArrayInputStream. Writing to a File will keep memory pressure low, but you'll incur the cost of writing to disk. Hope this helps!

[Siddhartha90]

The question is how does one convert an OutputStream to an InputStream. Alternatives are to write to a temp File or buffer it all in-memory.

There is a third option, which avoids using double-memory - https://stackoverflow.com/a/36961688/792238.