Do I have to pass tests?

Question

Please note: I am not asking for your opinion. I ask about agreements.

I'm just wondering if I should pass both passing and unsuccessful tests with the appropriate method names, such as Should_Fail_When_UsageQuantityIsNegative() , Should_Fail_When_UsageQuantityMoreThan50() , Should_Pass_When_UsageQuantityIs50() .

Or, instead, should I encode them for passing and keep all tests in the Passed state?

Thanks! Hooray!

Answer 1

When creating unit tests, they all must pass. This does not mean that you should not check for “failed” cases. It just means that the test should pass when it "fails."

Thus, you do not need to go through (preferably) a large number of tests and manually verify that the correct ones were passed and failed. This greatly violates the goal of automation.

As Mark Rottwevel notes in the comments, just checking that something failed is not always enough. Verify that the failure is the correct failure. For example, if you use error codes, and error_code equal to 0 indicates success, and you want to make sure that there is a failure, do not check this error_code != 0 ; instead, check, for example, that error_code == 19 or something that matches the correct error code.

Edit

There is one more point that I would like to add. Although the final version of your code that you are deploying should not have unsuccessful tests, the best way to ensure that you are writing the correct code is to write your tests before writing the rest of the code. Before you make any changes to the source code, write a unit test (or, ideally, a few unit tests) that should fail (or not compile) now, but pass it after your change has been made. This is a good way to make sure that the tests you write verify the right thing. So, to summarize, your final product should not have a rejection of unit tests; however, the software development process should include periods when you have already written unit tests that have not yet passed.

Answer 2

You should not have failed tests if your program does not act as if it were not intended.

If the intended behavior of your program is that something is crashing and it fails, this should trigger a test.

If the program passes in the place where it should fail, the test for this part of the code should fail.

Thus, the program does not work properly if all the tests fail.

Answer 3

You should never have fail tests, as others have pointed out, this violates the goal of automation. What you might want are tests that check your code, as expected when the inputs are incorrect. Considering your examples Should_Fail_When_UsageQuantityIsNegative() is a test that should pass, but the statements you make depend on what fail means. For example, if your code should throw an IllegalArgumentException when the usage is negative, then you might have a test like this:

 @Test(expected = IllegalArgumentException.class) public void Should_Fail_When_UsageQuantityIsNegative() { // code to set usage quantity to a negative value } 

Answer 4

There are several different ways to interpret the question if the tests fail.

A test similar to Should_Fail_When_UsageQuantityMoreThan50() should be a passing test that checks that the corresponding error is selected. Throws_Exception_When_UsageQuantityMoreThan50() or the like. Many test suites have special capabilities for testing exceptions: JUnit is an expected parameter and Perl modules such as Test :: Exception can even check warnings .

Tests must fail during the development process, which means that they are doing their job. You should be suspicious of a test suite that will never work, probably has poor coverage. Failure tests will be used to change the behavior of the public, errors and other errors of the developer or tests or code. But when they are done and pushed, the tests should be returned to pass.

Finally, there are legitimate cases where you have a known bug or missing feature that currently cannot be fixed or implemented. Sometimes errors are fixed by accident, so it's a good idea to write a test for it. When it passes, you know that the error is fixed, and you want to receive a notification when it starts to pass. Various test systems allow you to write tests that are expected to fail, and will only be visible if they pass. In Perl, this is TODO or the expected failure . POSIX has a number of results, such as UNRESOLVED, UNSUPPORTED, and UNTESTED , to cover this case.