Xcode 5 says tests failed, but green checkmarks appear

Question

I wrote some unit tests in the new XCTest Xcode 5 framework.

The tests seem to succeed most of the time, but I came across strange behavior when, if I run the tests enough times in a row, in the end I will get the message “test failed” and the console will show Program ended with exit code 0 , but before all tests there are green ticks.

What is wrong here?

Answer 1

This is definitely a mistake. Send the report via https://bugreport.apple.com/ do not hesitate to link to my report (15251050).

Answer 2

Until the error is fixed, close the project and open it.

Answer 3

Build Clean worked for me ... sometimes anyway. Product → Cleanliness.

Answer 4

If you look at the output console, you should notice that Xcode launches a test suite:

 Test Suite 'SomeClassTests' started at 2013-11-30 23:19:34 +0000 

but suddenly exits without running test cases:

 Program ended with exit code: 0 

and shows a notification that the test has failed.

Restarting Xcode helps temporarily as the problem returns quickly.

I stopped the tests in Xcode and ran all the tests on the command line using xctool .

Answer 5

For me, I think this problem was due to an incorrect OCMock configuration. Removing OCMock (completely) and re-adding it to the project in exactly the same way as the recommended OCMock seems to fix it.

A few specific things that I did wrong:

• I used #import "OCMock.h instead of #import "OCMock/OCMock.h . Xcode autocompleted the first.
• I had an unsuccessful link in the library search path from a previous attempt to install OCMock with a different directory structure.
• This is stupid, but trying to debug this problem, I commented on an instance of my OCMockObject in my setup method .. and did not forget to uncomment it. Nile OCMockObject caused a similar behavior, as described in other answers here. This is strange. In my current project, I have two tests that use the OCMockObject created in my setup. Both must fail. However, one of them passes, and the only difference that I see between them is that the one that passes by passes legally ...

In any case, when OCMock is configured correctly, my tests work the way they should

Answer 6

As soon as I came across this in my updated Xcode 5, I decided to share my similar permissions.

I notice that if I hit any breakpoint when starting the test and just continue to pass it as a transfer, it is very strange and unacceptable to run the whole package ...

I also noticed that adding hibernation, as mentioned above, as well as doing a clean build before running the tests, seems to solve the problem. However, creating a new test often repeats the problem, at least for this new test. Hope Apple fixes this soon!

Answer 7

This seems to be fixed in Xcode 5.1.1

Answer 8

This bug was fixed in Xcode 5.1:

Xcode 5.1 Release Notes

Notes specifically indicate:

Test target [test name] encountered an error (test process completed with code -1)

Attempted recovery by stopping work and restarting the simulator. (15929053)

Answer 9

Although I use logic tests, Xcode wants the iPhone simulator to work. Once in agony, trying to run the tests again, I added reset the contents of the iPhone simulator and the settings from it, and it helped.