Detecting dead tests and hard-coded data against limited non-determinism

Question

For those who are not sure what is meant by “limited non-determinism,” I recommend the Mark Seeman post .

The idea is a test that has deterministic values ​​only for data that affects the behavior of SUT. Non-relevant data may be somewhat random.

I like this approach. The more data is abstract, the more clear and expressive expectations become, and it really becomes more difficult to subconsciously select data for a test.

I am trying to “sell” this approach (along with AutoFixture ) to my colleagues, and yesterday we discussed this for a long time.
They suggested an interesting argument that it is unstable to debug tests because of random data.
At first it seemed a little strange, because, since we all agreed that the flow affecting the data should not be random, and this behavior is impossible. Nevertheless, I took a break to fully reflect on this problem. And I finally came to the following problem:

But some of my assumptions:

• The test code MUST be considered a production code.
• The test code MUST express the correct expectations and characteristics of the behavior of the system.
• Nothing warns you about inconsistencies better than a broken assembly (either not compiled or just failed tests - closed registration).

consider these two variants of the same test:

[TestMethod] public void DoSomethig_RetunrsValueIncreasedByTen() { // Arrange ver input = 1; ver expectedOutput = input+10; var sut = new MyClass(); // Act var actualOuptut = sut.DoeSomething(input); // Assert Assert.AreEqual(expectedOutput,actualOutput,"Unexpected return value."); } /// Here nothing is changed besides input now is random. [TestMethod] public void DoSomethig_RetunrsValueIncreasedByTen() { // Arrange var fixture = new Fixture(); ver input = fixture.Create<int>(); ver expectedOutput = input+10; var sut = new MyClass(); // Act var actualOuptut = sut.DoeSomething(input); // Assert Assert.AreEqual(expectedOutput,actualOutput,"Unexpected return value."); } 

Until now, God, everything works, and life is beautiful, but then the requirements change, and DoSomething changes its behavior: now it increases the input only if it is less than 10 and multiplied by 10 otherwise. What's going on here? A test with hard-coded data passes (almost by accident), while the second test sometimes fails. And both of them mistakenly cheat on tests: they test non-existent behavior.

It does not seem to matter if the data is hard-coded or random: it is simply inappropriate. And yet we have no reliable way to detect such "dead" tests.

So the question is:

Does anyone have any good advice on how to write tests on how such situations do not appear?

Answer 1

"then changes in requirements and DoSomething change their behavior"

It's true? If DoSomething modifies behavior, it violates the Open / Closed Principle (OCP). You may decide not to care about this, but it is closely related to why we trust the tests .

Every time you change existing tests, you reduce their reliability. Each time you change an existing production behavior, you will need to look at all the tests that relate to this production code. Ideally, you should visit each such test and briefly modify the implementation to see that it still fails if the implementation is incorrect.

For small changes, this may be practical, but for moderate changes, it would be wiser to stick with OCP: do not modify existing behavior; add new behavior side by side and let the atrophy of the old behavior.

In the example above, it may be clear that the AutoFixture test may be nondeterministically incorrect, but at a more understandable level, it is entirely possible that if you change production behavior without checking the tests, some tests can silently turn into false negatives . This is a common unit testing problem and is not specific to AutoFixture.

Answer 2

In this sentence, the answer is hidden:

[..], then changes in requirements and DoSomething change their behavior [..]

It would be easier if you do this:

• Modify expectedOutput first to meet the new requirements.
• Follow a failed test - it is important that it does not pass.
• Only then change DoSomething to meet the new requirements - retest the test.

This methodology is not related to a specific instrument, such as AutoFixture, it is only Test-Driven Development.

---

Where is AutoFixture really useful? Using AutoFixture, you can minimize part of the Arrange procedure.

Here's the original test, written idiomatically using AutoFixture.Xunit :

 [Theory, InlineAutoData] public void DoSomethingWhenInputIsLowerThan10ReturnsCorrectResult( MyClass sut, [Range(int.MinValue, 9)]int input) { Assert.True(input < 10); var expected = input + 1; var actual = sut.DoSomething(input); Assert.Equal(expected, actual); } [Theory, InlineAutoData] public void DoSomethingWhenInputIsEqualsOrGreaterThan10ReturnsCorrectResult( MyClass sut, [Range(10, int.MaxValue)]int input) { Assert.True(input >= 10); var expected = input * 10; var actual = sut.DoSomething(input); Assert.Equal(expected, actual); } 

In addition to xUnit.net, there is support for NUnit 2 .

NTN