How to copy InputStream to AsynchronousFileChannel

Question

I want to read from (Sercat servlet) InputStream and copy (large) content to a file asynchronously using AsynchronousFileChannel. I can do this with the usual FileChannel and read about the missing migration . But if I use Java 7 AsyncFileChannel, I always get a BufferOverflowException.

try (AsynchronousFileChannel output = AsynchronousFileChannel.open(path, StandardOpenOption.CREATE, StandardOpenOption.WRITE); output.lock(); // need to lock, this is one key reason to use channel ReadableByteChannel input = Channels.newChannel(inputStream); // servlet InputStream ByteBuffer buf = ByteBuffer.allocate(4096); int position = 0; int count; Future<Integer> lastWrite = null; while ((count = input.read(buf)) >= 0 || buf.position() > 0) { logger.info("read {} bytes", count); buf.flip(); output.write(buf, position); if (count > 0) position += count; buf.compact(); } if (lastWrite != null) lastWrite.get(10, TimeUnit.SECONDS); 

then at startup i get

 14:12:30.597 [http-bio-9090-exec-3] INFO cbpcBlobUploadServlet - read 4096 bytes 14:12:30.597 [http-bio-9090-exec-3] INFO cbpcBlobUploadServlet - read 0 bytes ... many more with 0 bytes read ... 14:12:30.597 [http-bio-9090-exec-3] INFO cbpcBlobUploadServlet - read 3253 bytes 14:12:30.605 [http-bio-9090-exec-3] ERROR cbpcBlobUploadServlet - null java.nio.BufferOverflowException: null at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183) ~[na:1.7.0_17] at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:393) ~[na:1.7.0_17] 

How can I fix BufferOverflow? Also, what is the correct way to pause the loop and wait for 0 bytes to be read?

Answer 1

Too late for the original poster, but anyway.

I tried to reproduce your problem (but with a slightly different sample, I duplicated a large file using channels):

 public static void main(String[] args) throws IOException, InterruptedException, ExecutionException { final InputStream inputStream = new FileInputStream("/home/me/Store/largefile"); final ReadableByteChannel inputChannel = Channels.newChannel(inputStream); final AsynchronousFileChannel outputChannel = AsynchronousFileChannel.open( FileSystems.getDefault().getPath( "/home/me/Store/output"), StandardOpenOption.CREATE, StandardOpenOption.WRITE); outputChannel.lock(); final ByteBuffer buffer = ByteBuffer.allocate(4096); int position = 0; int recievedBytes = 0; Future<Integer> lastWrite = null; while ((recievedBytes = inputChannel.read(buffer)) >= 0 || buffer.position() != 0) { System.out.println("Recieved bytes: " + recievedBytes); System.out.println("Buffer position: " + buffer.position()); buffer.flip(); lastWrite = outputChannel.write(buffer, position); // do extra work while asynchronous channel is writing bytes to disk, // in perfect case more extra work can be done, not just simple calculations position += recievedBytes; // extra work is done, we should wait, because we use only one buffer which can be still busy if (lastWrite != null) lastWrite.get(); buffer.compact(); } outputChannel.close(); inputChannel.close(); inputStream.close(); } 

In each iteration of the loop, we read a piece of data from the input stream, then we “push” this fragment into the output stream. The current thread does not wait for the completion of writing, it continues, so we can perform additional work. But before the new iteration, we must wait until the writing is complete. Try to comment

if (lastWrite != null) lastWrite.get();

and you will get

java.nio.BufferOverflowException .

Your code gave me a hint for using Future to handle the last write operation. But you missed the wait for the last operation.

In addition, I skipped the additional setting, which is in your fragment (just for simplicity, no additional setting is required when working with files).