Upload file or InputStream to S3 with progress callback

Question

We download the file using the Amazon AWS Java library and have difficulty getting the download. We are currently invoking the following:

File file = new File(localAsset.getVideoFilePath()); PutObjectRequest putObjectRequest = new PutObjectRequest(bucket, localAsset.getFileName(), file); s3.putObject(putObjectRequest); 

How can we set a callback to check file upload progress?

thanks

Answer 1

I ran into this exact problem and wrote a simple InputStream wrapper that outputs good progress indicators:

 import java.io.IOException; import java.io.InputStream; import org.apache.commons.vfs.FileContent; import org.apache.commons.vfs.FileSystemException; public class ProgressInputStream extends InputStream { private final long size; private long progress, lastUpdate = 0; private final InputStream inputStream; private final String name; private boolean closed = false; public ProgressInputStream(String name, InputStream inputStream, long size) { this.size = size; this.inputStream = inputStream; this.name = name; } public ProgressInputStream(String name, FileContent content) throws FileSystemException { this.size = content.getSize(); this.name = name; this.inputStream = content.getInputStream(); } @Override public void close() throws IOException { super.close(); if (closed) throw new IOException("already closed"); closed = true; } @Override public int read() throws IOException { int count = inputStream.read(); if (count > 0) progress += count; lastUpdate = maybeUpdateDisplay(name, progress, lastUpdate, size); return count; } @Override public int read(byte[] b, int off, int len) throws IOException { int count = inputStream.read(b, off, len); if (count > 0) progress += count; lastUpdate = maybeUpdateDisplay(name, progress, lastUpdate, size); return count; } static long maybeUpdateDisplay(String name, long progress, long lastUpdate, long size) { if (Config.isInUnitTests()) return lastUpdate; if (size < B_IN_MB/10) return lastUpdate; if (progress - lastUpdate > 1024 * 10) { lastUpdate = progress; int hashes = (int) (((double)progress / (double)size) * 40); if (hashes > 40) hashes = 40; String bar = StringUtils.repeat("#", hashes); bar = StringUtils.rightPad(bar, 40); System.out.format("%s [%s] %.2fMB/%.2fMB\r", name, bar, progress / B_IN_MB, size / B_IN_MB); System.out.flush(); } return lastUpdate; } } 

(This is copied and pasted from live code, so you may need to make a few corrections to make it work in your own code.)

Then just use the InputStream input method (be sure to specify the size!) And it will make a good progress bar for you. If you need the right callback, that will be pretty easy to do too.

Answer 2

There is currently no easy way to do this “inline,” but you can easily implement it by wrapping the InputStream to tell how far it reads.

You can read the team’s official answer to this question in their forums.

It can also be noted that the .NET SDK has this function, although, as far as I know, it is implemented in the same "hacker" way (just the S3 API itself really does not have built-in provisions for this elegantly). If you run into problems with its implementation, this may be the place where you can find inspiration.

Answer 3

Using the progress listener to track progress

 File file = new File(localAsset.getVideoFilePath()); PutObjectRequest putObjectRequest = new PutObjectRequest(bucket, localAsset.getFileName(), file); putObjectRequest.setGeneralProgressListener(new ProgressListener() { long transferred = 0; @Override public void progressChanged(ProgressEvent progressEvent) { transferred += progressEvent.getBytesTransferred(); Timber.i("Transferred %f%%", code, ((float)transferred / (float) fileSize) * 100f); } }); s3.putObject(putObjectRequest);