Allow Support for Uploading Byte Arrays and Strings in S3 TransferManager

[pradyuman]

Currently, the only way to upload data to S3 via TransferManager is through an InputStream or a File. I'm creating a service needs to upload data that is in memory. This means I need to write that Array[Byte] to a temp file, and then upload that temp file which introduces other variables that need to be tuned for performance (now need to make sure the temporary file system is optimized for our use case). It would be great if I could just pass in the byte array (can also convert to string if that's easier). This would mean I could keep everything in memory and would keep my performance benchmarking and production environment tuning simpler.

I'm sure this is a use case many other people may run into (or have already run into) so I think it's definitely worth taking a look at.

[pradyuman]

To give some context, the application ingests information using Kafka/Spark Streaming so I operate on an RDD to get my byte arrays. I then want to upload those byte arrays to S3. The application would operate completely in memory if I didn't need to create a tempfile (which also means the application would be more portable because I only have to worry about memory on my production environment).

[dagnir]

Hi @pradyuman, would wrapping your byte array in a ByteArrayInputStream and using PutObjectRequest#withInputStream work for you? Note that using an InputStream means the TransferManager won't be able to to parallelize the upload but hopefully these arrays are relatively small since they're kept in memory.

[dagnir]

The SDK also has StringIputStream if you want to convert the arrays to strings.

[pradyuman]

Those are both good options, but the arrays are 100-200MB in size so I imagine that I would benefit from the multipart optimizations.

[mzapletal]

I would be happy for an in-memory version of TransferManager as well - we have the requirement of downloading multiple (small) files concurrently which we keep in memory. However, it would be nice to use the TransferManager to benefit from its thread management.

[rmilejcz]

I would love to see this implemented, has there been any progress?

[shorea]

No progress has been made. We are considering supporting InputStreams in TransferManager (which would allow for other types like strings and byte arrays easily) but that's quite a ways off. aws/aws-sdk-java-v2#139

For now we recommend using an overload of put or get object (one takes/returns a string) and submitting to an executor if you need parallelization.

[debora-ito]

Hey @pradyuman @mzapletal @rmilejcz,

the SDK team has reviewed the feature request list for V1, and since they're concentrating efforts on V2 new features they decided to not implement this one in V1. It's still being considered for the TransferManager refactor in V2, see the referenced issue above. I'll go ahead and close this one.

Please feel free to comment on the V2 issue with your use case, and reach out if you have further questions.