The easiest way to ignore / write subprocess output with Java

Question

A subprocess in java is very expensive. Each process typically supports a NUMBERS stream.

• thread for process hosting (by JDK 1.6 on Linux)
• read stream to read / print / ignore input stream
• another thread to read / print / ignore the error stream
• more threads to timeout and monitor and kill the subprocess with your application
• business logic thread supporting subprocess return.

The number of threads gets out of control if there is a pool of threads that fake a subprocess to perform tasks. As a result, there may be more double parallel filament at the peak.

In many cases, we process the process only because no one can write a JNI to call our own function, which is not in the JDK (for example, chmod, ln, ls), runs a shell script, etc. etc.

Some threads can be saved, but some threads must be started to prevent the worst case (buffer overflow on the input stream).

How to reduce the overhead of creating a subprocess in Java to a minimum? I think that NIO processes the thread, combines and exchanges threads, reduces the priority of the background thread, reusing the process. But I have no idea if this is possible or not.

Answer 1

JDK7 will address this issue and provide a new redirectOutput / redirectError API in ProcessBuilder to redirect stdout / stderr.

However, the bad news is that they forget to provide "Redirect.toNull", which means you want to do something like "if (* nix) / dev / null elsif (win) nil"

It's unbelievable that the NIO / 2 api for the process is still missing; but I think redirectOutput + NIO2 AsynchronizeChannel will help.

Answer 2

I created an open source library that allows non-blocking I / O between java and child processes. The library provides an event-driven callback model. It depends on the JNA library for using native platform APIs such as epoll for Linux, kqueue / kevent on MacOS X, or IO Completion Ports on Windows.

The project is called NuProcess and can be found here:

https://github.com/brettwooldridge/NuProcess

Answer 3

To answer your topic (I don’t understand the description), I assume that you mean the output of the shell subprocess, check these SO problems:

platform independent / dev / null receiver for Java

Is there a Null OutputStream in Java?

Or you can close stdout and stderr for a command executed on Unix:

command > /dev/null 2>&1 

Answer 4

nio will not work, because when you create a process, you can only access the OutputStream, not the channel.

You can have 1 stream reading multiple InputStreams.

Sort of,

 import java.io.InputStream; import java.util.List; import java.util.concurrent.CopyOnWriteArrayList; class MultiSwallower implements Runnable { private List<InputStream> streams = new CopyOnWriteArrayList<InputStream>(); public void addStream(InputStream s) { streams.add(s); } public void removeStream(InputStream s) { streams.remove(s); } public void run() { byte[] buffer = new byte[1024]; while(true) { boolean sleep = true; for(InputStream s : streams) { //available tells you how many bytes you can read without blocking while(s.available() > 0) { //do what you want with the output here s.read(buffer, 0, Math.min(s.available(), 1024)); sleep = false; } } if(sleep) { //if nothing is available now //sleep Thread.sleep(50); } } } } 

You can associate the above class with another class that is waiting for processes to finish, something like

 class ProcessWatcher implements Runnable { private MultiSwallower swallower = new MultiSwallower(); private ConcurrentMap<Process, InputStream> proceses = new ConcurrentHashMap<Process, InputStream>(); public ProcessWatcher() { } public void startThreads() { new Thread(this).start(); new Thread(swallower).start(); } public void addProcess(Process p) { swallower.add(p.getInputStream()); proceses.put(p, p.getInputStream()); } @Override public void run() { while(true) { for(Process p : proceses.keySet()) { try { //will throw if the process has not completed p.exitValue(); InputStream s = proceses.remove(p); swallower.removeStream(s); } catch(IllegalThreadStateException e) { //process not completed, ignore } } //wait before checking again Thread.sleep(50); } } } 

In addition, you do not need to have 1 stream for each error stream, if you use ProcessBuilder.redirectErrorStream (true) and you do not need 1 stream to read the process input stream, you can simply ignore the input if you are not writing anything.

Answer 5

You do not need additional threads to run the subprocess in java, although handling timeouts makes the situation a bit more complicated:

 import java.io.IOException; import java.io.InputStream; public class ProcessTest { public static void main(String[] args) throws IOException { long timeout = 10; ProcessBuilder builder = new ProcessBuilder("cmd", "a.cmd"); builder.redirectErrorStream(true); // so we can ignore the error stream Process process = builder.start(); InputStream out = process.getInputStream(); long endTime = System.currentTimeMillis() + timeout; while (isAlive(process) && System.currentTimeMillis() < endTime) { int n = out.available(); if (n > 0) { // out.skip(n); byte[] b = new byte[n]; out.read(b, 0, n); System.out.println(new String(b, 0, n)); } try { Thread.sleep(10); } catch (InterruptedException e) { } } if (isAlive(process)) { process.destroy(); System.out.println("timeout"); } else { System.out.println(process.exitValue()); } } public static boolean isAlive(Process p) { try { p.exitValue(); return false; } catch (IllegalThreadStateException e) { return true; } } } 

You can also play with reflection, as in Can I read from an InputStream with a timeout? to get the NIO FileChannel from Process.getInputStream() , but then you have to worry about different versions of the JDK in exchange for getting rid of the poll.

Answer 6

Since you mention chmod , ln , ls and shell scripts, it looks like you are trying to use Java to program the shell. If so, you may want to consider another language that is better suited for this task, such as Python, Perl, or Bash. Although, of course, it is possible to create subprocesses in Java, interact with them through their standard input / output / error streams, etc., I think you will find that the scripting language makes this kind of code less verbose and easier to maintain than Java

Answer 7

Can you use JNA to record your own calls?

See the answer here. How do I programmatically change file permissions? for a good example of chmod.

Much easier than JNI, and much faster than subprocesses!

Answer 8

Have you considered using one long-term helper process written in another language (perhaps a shell script?) That will consume commands from java via stdin and perform file operations in response?