Knowledge on passing the test @ Ignore'd

Question

In a small open source project written in Java, we get a lot of error reports, fewer unit test for these error reports, and fewer patches that cover these errors.

When a unit test is provided, but without a patch, we check for an error regarding the trunk by adding a test function to the set. We fix it, but add an annotation to it junit4 @Ignoreso as not to break the assembly for a known error.

Some errors may be related, or a change made for one error may fix another. We would like to know when the previous failed test case no longer works, so we can notify people who are observing this error and close the error.

Let's say we have some buggy function - an exception occurs, has unwanted side effects or returns the wrong value. ( Real world example )

public void buggyFunction() {
    throw new UnsupportedOperationException("this will be implemented later");
}

And some unit test for testing buggyFunction

@Test
public void knownIssue() {
    buggyFunction();
}

Option 1 : @Ignoretest

@Ignore("this test is currently failing. see bug 12345")
@Test
public void knownIssue() {
    buggyFunction();
}

Option 2 . The standard way junit4 is to use @Test(expected=MyException.class)or sprinkle @Rule ExpectedExceptionthroughout the test function. Also, do not tell the user a useful message about why a failed test means that the error is fixed, and update the unit test and close the error. In addition, the test passes if the expected exception is selected, but the test does not make sense in this case. It would be better to either crash (when the error is fixed) or skip (when the error is still open).

// when this bug is fixed, it should not throw an exception
// TODO: delete expected=UnsupportedOperationException.class
@Test(expected=UnsupportedOperationException.class)
public void knownIssue() {
    buggyFunction();
}

OR

@Rule
public final ExpectedException thrown = ExpectedException.none();

@Test
public void knownIssue() {
    thrown.expect(UnsupportedOperationException.class);
    thrown.expectMessage("this will be implemented later");
    buggyFunction();
    thrown.expect(ExpectedException.none());
}

Option 3 : drill the test using the boiler plate

@Test
public void knownIssue() {
    try {
        buggyFunction();
    } catch (UnsupportedOperationException e) {
        // we know that buggyFunction is broken, so skip this test
        assumeTrue("Skipping test. Expected exception: " + e, false);
    }
    // surprise! buggyFunction isn't broken anymore!
    fail("The function is no longer buggy! " +
         "Update the unit test and close bug 12345!");
}

, :

• , .
• , .
• , .
• , hamcrest

- Python, . Java 6 (, , ), , , , 3. , , .

def alertWhenFixed(expected=Exception, bug=12345):
    def decorator(func):
        def func_wrapper(*args, **kwargs):
            try:
                func(*args, **kwargs)
            except Exception as e:
                if isinstance(e, expected):
                    assumeTrue("Skipping test. Expected exception: {}"
                               .format(e), false)
                else:
                    raise e
            fail("The function is no longer buggy! " +
                 "Update the unit test and close bug {}".format(bug))
        return func_wrapper
    return decorator

@alertWhenFixed(expected=UnsupportedOperationException, bug=12345)
def knownIssue():
    buggyFunctionThrowsException()

@alertWhenFixed(expected=AssertionFailed)
def knownIssue():
    assertEquals(42, buggyFunctionReturnsWrongValue())

@alertWhenFixed(expected=AssertionFailed)
def knownIssue():
    buggyFunctionHasWrongSideEffect()
    assertEquals(42, getSideEffect())

, , .

100% , - / , , , , , .

, Java 6 7?

Answer 1

TestSuite , , (!).

/**
 * @see ...original now ignored test
 */
@Test
public void defect123124() {
    expectThrows(UnsupportedOperationException.class, () -> buggyFunction());
}

" ", , . , . @Ignore.

. , - , .

Answer 2

, , .

, : , . , , , , buggyFunction(), . :

. , :

• .
• : !

: , . buggyFunction, , , , .

, , , , . , . , . - , .

, - 2 3. "", : - , . , , ; , . (, , ); , ( ).

Answer 3

Testsuites , JUnit Category. (, , . . .)

:

public interface KnowIssue {
    // JUnit category marker interface
}

@Ignore ( ):

@Test
@Category(KnowIssue.class)
public void myFailingTest() {
    // some test
}

(-) KnownIssue:

@RunWith(Categories.class)
@ExcludeCategory(KnownIssue.class) // <- exclude known issues!
@Suite.SuiteClasses({ /** your Testclasses here */ })
public class AllMyGoodTests {
    // test suite
}

, , (), , - () :

@RunWith(Categories.class)
@IncludeCategory(KnownIssue.class) // <- only run known issues!
@Suite.SuiteClasses({ /** your Testclasses here */ })
public class KnownIssueTests {
    // test suite
}

"" .

, ( ↔ ) ( @RunWith(KnownIssueCategoryRunner.class), .