Can I make unit test inconclusive if the unit test props fail?

Question

Consider unit testing a dictionary object. The first unit tests you can write are a few that simply add items to the dictionary and check for exceptions. The next test may be something like checking that the counter is accurate, or that the dictionary returns the correct list of keys or values.

However, each of these later cases requires the dictionary to be able to reliably add elements first. If the tests that add elements fail, we have no idea whether our later tests are incorrect because they are testing, or because the assumption that we can reliably add elements is wrong.

Can I declare a set of unit tests that lead to the fact that this unit test will be inconclusive if any of them fails? If not, how would I best get around this? I set up my unit tests incorrectly, what did I encounter with this predicament?

Answer 1

It is not as difficult as it might seem. We rephrase the question a bit:

If I test my piece of code that requires System.Collections.Generic.List<T>.Add , what should I do when one day Microsoft decides to break .Add to List<T> ? Can I do my tests depending on this to work unconvincing?

The answer to this is obvious; you are not . You allow them to fail for one simple reason - your assumptions have failed, and the test should fail. This is the same here. After you receive tests for work, from now on you can add work. You should not treat your tested code differently than the tested third-party code. Once this is proven, you think it is true.

In another note, you can use a concept called defensive statements . In your test for deletion after the arrangement phase, you introduce an additional assert phase that checks your initial assumptions (in this case, that the addition works). More information about this method can be found here .

To add an example, NUnit uses the concept above disguised as Theory . This does exactly what you suggested (but it seems to have more to do with data-based testing, rather than general utility):

The theory itself is responsible for ensuring that all data provided is consistent with its assumptions. He does this with the Assume.That (...) construct, which works the same as Assert.That (...) but does not fail . If the assumption is not satisfied for a specific test case, this case returns a non-convertible result , not success or failure.

However, I think that Mark Seemann points out in the answer to the question I linked, it makes the most sense:

There may be many prerequisites that must be met for this test case, so you may need more than one Guard statement. Instead of repeating these tests in all tests, having one (and only) test for each precondition makes your test code more suitable, since you will have fewer repetitions in this way.

Answer 2

Good question, I often think about it and had this problem the other day. What I did is how the basics of our collection work using the vocabulary behind the scenes. For instance:

 public class MyCollection { private IDictionary<string, int> backingStore; public MyCollection(IDictionary<string, int> backingStore) { _backingStore = backingStore; } } 

Then we check the implementation of the add. Since we had a dictionary by reference, we could argue that after adding the elements, our business logic was correct.

For example, the pseudo code for the addon was something like this:

 public void Add(Item item) { // Check we have not added before // More business logic... // Add } 

Then one could write a test:

 var subject = new MyCollection(backingStore); subject.Add(new Item()) Assert.That(backingStore.Contains(itemThatWeAdded) 

Then we continued to supplant other methods, such as search and delete.

Your question is what you should do with regard to hacking addition, in turn, interrupting the search. This is a trick scenario 22. Personally, I would rather cut out a back-up store and use it as an implementation detail. So that’s what we did. We reorganized the tests for using the system under test, and not for the backup storage for claims. The great thing is that the backup storage is public initially, it allows you to test small parts of the code, rather than performing both adding and searching at a time.

The test to be added then looked like this after we reorganized the collection so as not to expose the backup storage.

 var subject = new MyCollection(); var item = new Item() subject.Add(item) Assert.That(subject.Has(item), Is.True); 

In this case, I think that everything is in order. If you cannot add items successfully, then you are sure that the devil cannot get anything because you did not add them. As long as your tests are called well, any developer who sees some tests, such as " CanOnlyAddUniqueItemsToCollection ", will point future developers in the right direction, in other words, the addition will be broken. Just make sure your tests are well-named and you should give as much help as possible.

Answer 3

I do not see this as too big a problem. If your dictionary class is not too large, and the unit test for this class is the only unit test testing this code, then when your add method is broken and several tests do not work, you still know that the problem is with the dictionary class and can identify it debug and easily fix it.

Where this becomes a problem, a problem arises when you have other code smells or design problems, such as:

• unit testing tests test many application classes, but using mocks can help here.
• unit tests are actually system tests that simultaneously create and test many application classes.
• The dictionary class is too large and complex, therefore, when it breaks, and tests do not allow to determine which part is broken.

Answer 4

It is very interesting. We use NUnit, and I can say that it runs the test methods in alphabetical order. It may be an overly artificial way to order your tests, but if you created your test classes so that you can select the pre-task methods in alphabetical / numerical names at first, you can do what you want.

I find that I am writing a test method, shooting only at him to see how he fails, and then wrote the code to get it through. When I’ve finished everything, I can run the whole class, and everything passes - it doesn’t matter in what order the tests passed, because everything "works" because I did incremental.

Now, if I break something in what I'm testing, who knows that everything will fail in harness. I think this does not matter much to me - I have a long list of failures, and I can tease what went wrong.