Who does test the tests?

A big Problem of testing has always been the reliability of the tests. Of course you can have 100% code coverage but does this make your code well testet? Are the tests really meaningful or just written because it hast to be done? Mutation Testing is a concept that tries to solve this problem. It does test the tests!

What are bad tests?

Let me give u a quick example

public class Mutation {
public static boolean dividableByThree(int number) {
return (number % 3 == 0);
}
}

I think its pretty obvious what this code is doing, so lets write a test for it.

public class MutationTest {
@Test
public void testDividableByThree(){
Mutation.dividableByThree(6);
}
}

Yes i know, this test is complete bullshit. Nevertheless it gives me a code coverage of 100% without giving us any security about our code.
For example we could change the code to

public class Mutation {
public static boolean dividableByThree(int number) {
return (number * 3 == 0);
}
}

Now the logic is a complete different one because we multiply, but the test ist still green. What we did is we mutated our code to see if our test will respond to that mutation. This is basically what Mutation Testing is about. You change your code and hope that the test will run red. Otherwise you either have a bad test or need to reconsider your implementation.

Mutation Testing

There are two types of outcome of a Mutation Test. Either your test survived  (still green) the Mutation or it got killed (red).
Of course it is very hard to alter your code and do the Mutation Testing on your own.  You can use Plugins like Pitclipse.

After you installed it you can right-click on your Tests and -> Run as -> PIT Mutation Tests

As you could have guessed all of our tests survived which is a bad thing. When we change the Test to

public class MutationTest {
@Test
public void testDividableByThree() {
Assert.assertTrue(Mutation.dividableByThree(6));
}
}

You can see that almost all the Mutation got killed.

Spread the love