How to write an S3 object to a file?

Question

What is the fastest way to write an S3 object (from which I have a key) to a file? I am using Java.

Answer 1

While IOUtils.copy() and IOUtils.copyLarge() great, I would prefer the old school path to go through the input stream until the input stream returns -1. What for? I used to use IOUtils.copy (), but there was a specific use case when I started downloading a large file from S3, and then for some reason, if this stream was interrupted, the download did not stop, and it continued and continued until while the whole file has been downloaded.

Of course, this has nothing to do with S3, it is just an IOUtils library.

So, I prefer this:

 InputStream in = s3Object.getObjectContent(); byte[] buf = new byte[1024]; OutputStream out = new FileOutputStream(file); while( (count = in.read(buf)) != -1) { if( Thread.interrupted() ) { throw new InterruptedException(); } out.write(buf, 0, count); } out.close(); in.close(); 

Note. It also means that you do not need additional libraries.

Answer 2

Since Java 7 (published back in July 2011), this is the best way: Files.copy() from java.util.nio.file .

Copies all bytes from the input stream to a file.

Thus, you do not need an external library and do not flip your own byte byte bytes. Two examples below, both of which use the input stream from S3Object.getObjectContent() .

 InputStream in = s3Client.getObject("bucketName", "key").getObjectContent(); 

1) Writing to a new file at the specified path:

 Files.copy(in, Paths.get("/my/path/file.jpg")); 

2) Write to the temp file in the system default location tmp:

 File tmp = File.createTempFile("s3test", ""); Files.copy(in, tmp.toPath(), StandardCopyOption.REPLACE_EXISTING); 

(Without specifying an option to replace an existing file, you will get a FileAlreadyExistsException .)

Also note that getObjectContent() Javadocs urges you to close the input stream :

If you are extracting S3Object, you should close this input stream as soon as possible, since the contents of the object are not buffered by the memory and stream directly from Amazon S3. In addition, failure to close this thread may block the request pool.

Thus, it should be safer to wrap everything in try-catch-finally and do in.close(); in the finally block.

The above assumes that you are using the official Amazon SDK ( aws-java-sdk-s3 ).

Answer 3

The AmazonS3Client class has the following method:

 S3Object getObject(String bucketName, String key) 

The returned S3Object has a ...

 java.io.InputStream getObjectContent() 

.. which receives the contents of the object as a stream. I would use IOUtils from Apache Commons like this:

IOUtils.copy(s3Object.getObjectContent(), new FileOutputStream(new File(filepath)));

Answer 4

How about this one liner using TransferManager:

 TransferManagerBuilder.defaultTransferManager .download("bucket-name", "key", new File("."))