Skip to content

Support Uploading to S3 using an OutputStream #1139

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sixcorners opened this issue Apr 29, 2017 · 8 comments
Closed

Support Uploading to S3 using an OutputStream #1139

sixcorners opened this issue Apr 29, 2017 · 8 comments
Assignees
Labels
guidance Question that needs advice or information. response-requested Waiting on additional info or feedback. Will move to "closing-soon" in 5 days.

Comments

@sixcorners
Copy link

Aren't we outputting to S3? Shouldn't we be getting an output stream to resources in S3 to write stuff?
spring-cloud-aws has an implementation of this here:
SimpleStorageResource

@kiiadi
Copy link
Contributor

kiiadi commented May 1, 2017

An InputStream allows the SDK to pull data into S3. What's your use case exactly? In a future version of the SDK we're looking at changing the streaming API a bit. It would be good to understand what you're trying to do.

@sixcorners
Copy link
Author

I guess I'm not doing anything that outputs to OutputStreams anymore so I don't have a specific need. But I still think it makes sense that APIs that let you write files should return OutputStreams.
The functionality that would actually be helpful to me would be a function that uploads streams (input or output streams) without buffering it to determine Content-Length like how that SimpleStorageResource class does it but I guess I can create a new issue for that.

@dagnir
Copy link
Contributor

dagnir commented Jun 26, 2017

Going to go ahead close this issue. @sixcorners we're currently working on the next iteration of the SDK which address some of these drawbacks so please be on the lookout for it!

@dagnir dagnir closed this as completed Jun 26, 2017
@martindvds
Copy link

This would be a highly appreciated feature, since it would make it a lot easier to plug S3 into places where you already write into a FileOutputStream, without having to redesign anything.

@123tharun
Copy link

123tharun commented Nov 9, 2018

Is this implemented yet?. @dagnir @kiiadi @sixcorners

We are generating documents using doc4jx which gives us an outputstream, we do not want to store in a document and upload it to the S3. We would directly like to store the inputstream from doc4jx directly to the S3.

@kvsravindrareddy
Copy link

I really appreciate if you could provide some examples to write output stream directly S3 bucket

@guerard
Copy link

guerard commented Dec 11, 2018

@kvsravindrareddy You can only provide an InputStream to the AmazonS3 client for it to read from. You'll need to know the length in bytes of the content prior to uploading if you want it to actually be streamed. AmazonS3 will otherwise just buffer the entire contents in-memory and then send it off.

If you do know how many bytes you're sending, then you could pipe streams e.g.:

long numBytes;  // length of data to send in bytes..somehow you know it before processing the entire stream
PipedOutputStream os = new PipedOutputStream();
PipedInputStream is = new PipedInputStream(os);
ObjectMetadata meta = new ObjectMetadata();
meta.setContentLength(numBytes);
new Thread(() -> {
    /* Write to os here; make sure to close it when you're done */
}).start();
amazonS3.putObject("myBucket", "myKey", is, meta);
is.close();  // always close your streams

The question is how does one convert an OutputStream to an InputStream. Alternatives are to write to a temp File or buffer it all in-memory. If you don't know the size, then you won't see any benefit to pipes w/ AmazonS3 compared to using ByteArrayOutputStream + ByteArrayInputStream. Writing to a File will keep memory pressure low, but you'll incur the cost of writing to disk. Hope this helps!

@srchase srchase added guidance Question that needs advice or information. needs-response and removed Question labels Jan 4, 2019
@debora-ito debora-ito added response-requested Waiting on additional info or feedback. Will move to "closing-soon" in 5 days. and removed needs-response labels Feb 25, 2020
@Siddhartha90
Copy link

Siddhartha90 commented Aug 25, 2021

The question is how does one convert an OutputStream to an InputStream. Alternatives are to write to a temp File or buffer it all in-memory.

There is a third option, which avoids using double-memory - https://stackoverflow.com/a/36961688/792238.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
guidance Question that needs advice or information. response-requested Waiting on additional info or feedback. Will move to "closing-soon" in 5 days.
Projects
None yet
Development

No branches or pull requests

10 participants