Ruby: reading big data from stdout and stderr of an external process in Windows

Question

Cheers, everyone

I need to run a potentially lengthy process from Ruby 1.9.2 on Windows, and then capture and parse data from an external output process and errors. A large amount of data can be sent for each, but I am only interested in one line at a time (without capturing and not saving all the output).

After a little research, I found that the Open3 class will take care of executing this process and giving me IO objects related to the standard output and the process error (via popen3 ).

 Open3.popen3("external-program.bat") do |stdin, out, err, thread| # Step3.profit() ? end 

However, I'm not sure how to constantly read from both threads without blocking the program. Since calling IO#readlines on out or err when a lot of data has been sent results in a memory allocation error, I try to constantly check both threads for accessible input, but not having much luck with any of my implementations.

Thanks in advance for any advice!

Answer 1

After many different trial and error attempts, I eventually came up with two streams: one for reading from each stream ( generator.rb is just a script that I wrote to output information to standard and erroneous):

 require 'open3' data = {} Open3.popen3("ruby generator.rb") do |stdin, out, err, external| # Create a thread to read from each stream { :out => out, :err => err }.each do |key, stream| Thread.new do until (line = stream.gets).nil? do data[key] = line end end end # Don't exit until the external process is done external.join end puts data[:out] puts data[:err] 

It simply prints the last line sent to standard output and error by the calling program, but, obviously, it can be expanded for additional processing (with different logic in each thread). The method that I used before I finally came up with this led to some crashes due to race conditions; I don't know if this code is still vulnerable, but I still experience a similar crash.