Monthly Archives: August 2011

Why System tests can’t replace Unit tests

Several times I have come across claims of redundant testing.

Most of which claim that a System test can cover all the functionality, so why do we need a unit test?

First of all I would like to emphasize their differences.

Unit tests

Unit tests are basically short and specific.

Checking your helper class returns the right results, test your algorithm on different inputs and checking all edge cases. It is the smallest and basic testable part in an application.

They test that the code actually does what it should… CODE WISE.

Due to their nature, they usually don’t need any third parties or other external dependencies. they are self contained.

System Tests

A complete and integrated testing to make sure all the different parts of the system work together as expected.

These tests are usually more complicated. Many of them will include third parties for their automation, they fall into the “black box testing” criteria and they usually test the functionality of the application as a whole.

Wouldn’t a system test be able to cover any unit test?

Absolutely not!

Think about it.

Even if the same data is created in the beginning of the test and the same assertions are made, how can you be certain who is responsible for the success or failure? there are usually several components, some of which might affect the result.

The test can fail due to several components which will require more investigation, drilling into each components in any case, making sure “this part up until here” works OK and will eventually lead to more wasted time asking yourself “why didn’t I write a unit test for this”.

If the test passed, does it guarantee that everything is OK?

Your first intuition would be YES. but… you’d be wrong 🙂

This only means the SYSTEM is OK at this point. maybe your code has an error which is luckily missed due to another component, ran after your code.

What will happen if that piece of code changes and the actual bugs appear? how much time will you spend tracing down the bug?

Moreover, system tests will usually cover main flows, not edge cases.

We can write system tests for all edge cases.

Yes, you CAN. but how effective will they be? how long will they run?

System tests are almost always slower then unit tests. if not, something is wrong with your unit tests 🙂

Secondly, as explained before, that will most likely violate the “black box testing” methodology since you will have to be aware of the edge cases IN THE CODE.

Another issue is catching bad commits. executing tests before committing will detect your errors.

No problem here, All our programmers execute tests before they commit.

Do they now?

As per my experience, most developers will run the minimal set of tests they can to make sure they did not break anything. They will search for all related unit tests using references to the changed functions or using some simple type searching. most of the time they will not be even aware you have system tests that cover the part they’ve changed.

Still, if we make sure everyone runs ALL the tests and we do not suffer from long execution times, why do we still need unit tests?
that’s simple, because of the “ignored test” syndrome.

never heard of it? c’mmon…

when a test is unstable or fails and you have no time to fix it, you just put it in ignore or ignore it anyway.

but, since this is a system test it checks several components, meaning you didn’t just disable that specific test but all other “disguised” tests for sub-components. had you written unit tests, this test had less effect on the questions: “should we fix it now? is it top priority? is anything else covering this issue?”

To conclude,

System tests and Unit tests each have a specific role. They complement each other.

If you are pressured, System tests will cover basic functionality. but if you have the time, do both, ALWAYS!

For more formal definitions look at:

Unit Tests

System Tests

Unit tests vs System tests