Java exec method, how to handle threads correctly

Question

What is the correct way to create and consume threads (IO) of an external process with Java? As far as I know, java end input streams (process output) should be consumed in threads parallel to the input process due to a possible limited buffer size.

But I'm not sure what I ultimately need to synchronize with these consumer threads, or is it just enough to wait for the process to complete using the waitFor method to be sure that the entire process output is actually consumed? IE is possible, even if the process terminates (closes the output stream), is there still unread data at the java end of the stream? How does waitFor really even know when the process will be completed? For the process in question, EOF (closing the java-end of its input stream) signals to him about the exit.

My current thread processing solution is running

 public class Application { private static final StringBuffer output = new StringBuffer(); private static final StringBuffer errOutput = new StringBuffer(); private static final CountDownLatch latch = new CountDownLatch(2); public static void main(String[] args) throws IOException, InterruptedException { Process exec = Runtime.getRuntime().exec("/bin/cat"); OutputStream procIn = exec.getOutputStream(); InputStream procOut = exec.getInputStream(); InputStream procErrOut = exec.getErrorStream(); new Thread(new StreamConsumer(procOut, output)).start(); new Thread(new StreamConsumer(procErrOut, errOutput)).start(); PrintWriter printWriter = new PrintWriter(procIn); printWriter.print("hello world"); printWriter.flush(); printWriter.close(); int ret = exec.waitFor(); latch.await(); System.out.println(output.toString()); System.out.println(errOutput.toString()); } public static class StreamConsumer implements Runnable { private InputStream input; private StringBuffer output; public StreamConsumer(InputStream input, StringBuffer output) { this.input = input; this.output = output; } @Override public void run() { BufferedReader reader = new BufferedReader(new InputStreamReader(input)); String line; try { while ((line = reader.readLine()) != null) { output.append(line + System.lineSeparator()); } } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } finally { try { reader.close(); } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } finally { latch.countDown(); } } } } } 

Is it necessary to use a latch here, or is waitFor all output used? Also, if the output does not end / contains a new line, will readLine skip the output or still read the rest? Does zero reading occur, does the process terminate at its end - is there another scenario in which zero could be read?

What is the correct way to handle threads, can I do something better than in my example?

Answer 1

waitFor indicates that the process has finished, but you cannot be sure that the threads that collect the lines from its stdout and stderr are also finished, so using a latch is a step in the right direction, but not optimal. Instead of waiting for the latch, you can wait for the threads directly:

 Thread stdoutThread = new Thread(new StreamConsumer(procOut, output)).start(); Thread stderrThread = ... ... int ret = exec.waitFor(); stdoutThread.join(); stderrThread.join(); 

By the way, storing strings in a StringBuffer is useless work. Instead, use an ArrayList<String> , put the strings there without any conversion, and finally retrieve them in a loop.

Answer 2

Your appapproach is right, but it's better to remove CountDownLatch and use ThreadPool, rather than directly creating a new thread. From ThreadPool you will receive two futures that you can wait for after completion.

But I'm not sure what I ultimately need to synchronize with these consumer threads, or is it just enough to wait for the process to complete using the waitFor method to be sure that the entire process output is actually consumed? IE is possible, even if the process terminates (closes the output stream), is there still unread data at the end of the java stream?

Yes, such a situation may arise. Terminating and reading I / O streams are unrelated processes.