Rspec: how to continue the test after the first failure

Question

I use rspec to test the system. The device is modular, so any number of sub-devices can be connected to a test installation. There are many places where I want to write tests that will go through connected sub-devices and run the same tests on each device.

Basically, this is what I am trying to do:

before(:all) @tool = discover_sub_devices() @tool.sub_devices.should == exp_sub_device_list end describe "some sub device tests" do @tool.sub_devices.each do |dev| it "should succeed at doing things" do dev.do_thing.should == success end end end 

Unfortunately this will not work. I get errors saying that @tool is nill and does not contain the sub_devices class before the tests even run. Thus, the tests are analyzed before the before(:all) block is run.

One way to make it work is to put a loop inside the it block. Like this:

 it "should succeed at doing things" do @tool.sub_devices.each do |dev| dev.do_thing.should == success  end end 

The problem with this is that I really want to test all the sub-devices, even if the first one does not work. I want to see a report on how many device failures were unsuccessful. This code breaks as soon as one crashes, rather than testing the rest.

I understand that this is probably not an ordinary use case for rspec, but it would be very convenient for our test situation if I could do this work.

Any suggestions?

Answer 1

Here are some ways to write this.

It is better to avoid using before :all . It is also better to avoid creating objects outside of the examples.

 describe "some sub device tests" do let(:tool) { discover_sub_devices } it "matches the sub device list" do tool.sub_devices.should be == expected_sub_devices_list end it "succeeds with all sub-devices" do failures = tool.sub_devices.reject{|d| d.do_thing == success} # option 1 failures.should be_empty # will show just a yes/no result # option 2 failures.size.should be == 0 # will show the number of failures # option 3 failures.should be == [] # will display all sub-devices which fail end end 

Answer 2

The problem you are facing is that the body of the describe block starts immediately, and the bodies of the let , before and it blocks are executed later.

Assuming you don't need to reopen devices every time, you can reorganize your code as follows, excluding the before call:

 describe "some sub device tests" do tool = discover_sub_devices() it "should discover sub devices correctly" do tool.sub_devices.should == exp_sub_device_list end tool.sub_devices.each do |dev| it "should succeed at doing things" do dev.do_thing.should == success end end end