Why System tests can’t replace Unit tests

Several times I have come across claims of redundant testing.

Most of which claim that a System test can cover all the functionality, so why do we need a unit test?

First of all I would like to emphasize their differences.

Unit tests

Unit tests are basically short and specific.

Checking your helper class returns the right results, test your algorithm on different inputs and checking all edge cases. It is the smallest and basic testable part in an application.

They test that the code actually does what it should… CODE WISE.

Due to their nature, they usually don’t need any third parties or other external dependencies. they are self contained.

System Tests

A complete and integrated testing to make sure all the different parts of the system work together as expected.

These tests are usually more complicated. Many of them will include third parties for their automation, they fall into the “black box testing” criteria and they usually test the functionality of the application as a whole.

Wouldn’t a system test be able to cover any unit test?

Absolutely not!

Think about it.

Even if the same data is created in the beginning of the test and the same assertions are made, how can you be certain who is responsible for the success or failure? there are usually several components, some of which might affect the result.

The test can fail due to several components which will require more investigation, drilling into each components in any case, making sure “this part up until here” works OK and will eventually lead to more wasted time asking yourself “why didn’t I write a unit test for this”.

If the test passed, does it guarantee that everything is OK?

Your first intuition would be YES. but… you’d be wrong 🙂

This only means the SYSTEM is OK at this point. maybe your code has an error which is luckily missed due to another component, ran after your code.

What will happen if that piece of code changes and the actual bugs appear? how much time will you spend tracing down the bug?

Moreover, system tests will usually cover main flows, not edge cases.

We can write system tests for all edge cases.

Yes, you CAN. but how effective will they be? how long will they run?

System tests are almost always slower then unit tests. if not, something is wrong with your unit tests 🙂

Secondly, as explained before, that will most likely violate the “black box testing” methodology since you will have to be aware of the edge cases IN THE CODE.

Another issue is catching bad commits. executing tests before committing will detect your errors.

No problem here, All our programmers execute tests before they commit.

Do they now?

As per my experience, most developers will run the minimal set of tests they can to make sure they did not break anything. They will search for all related unit tests using references to the changed functions or using some simple type searching. most of the time they will not be even aware you have system tests that cover the part they’ve changed.

Still, if we make sure everyone runs ALL the tests and we do not suffer from long execution times, why do we still need unit tests?
that’s simple, because of the “ignored test” syndrome.

never heard of it? c’mmon…

when a test is unstable or fails and you have no time to fix it, you just put it in ignore or ignore it anyway.

but, since this is a system test it checks several components, meaning you didn’t just disable that specific test but all other “disguised” tests for sub-components. had you written unit tests, this test had less effect on the questions: “should we fix it now? is it top priority? is anything else covering this issue?”

To conclude,

System tests and Unit tests each have a specific role. They complement each other.

If you are pressured, System tests will cover basic functionality. but if you have the time, do both, ALWAYS!

For more formal definitions look at:

Unit Tests

System Tests

Unit tests vs System tests

2 thoughts on “Why System tests can’t replace Unit tests

  1. ripper243

    Nice post!

    Too bad you didn’t talk about integration tests at all, they’re an important and sometimes overlooked class of tests.

    Often, I find that integration tests provide more overall ROI than unit tests. With pure unit tests, you have to mock out everything, which often leads to complicated mock setup. Integration tests can usually test much the same flow as unit tests, and often be just as quick as unit tests, or almost the same speed.

    You’re still missing out the purity and isolation of unit tests, but you gain much more confidence your actual code works in the real world, plus they are sometimes easier to write. Of course in an ideal world you’ll have unit, integration and system tests all together, but in the real world we often sacrifice one, two or sometimes three elements from this list. Just make real ROI the guiding principal behind every commit and every test you write, and you’ll do fine.

  2. Gilly

    A good metaphor to describe the difference between unit tests and system tests is to think of it as testing your car…

    When going to a mechanic, running a system test is like trying to drive and seeing that the car doesn’t work. What’s wrong ? you’ll never know. All you will know is that something is wrong and that the car doesn’t work anymore.

    Running unit tests on the other hand, is like taking apart the car and motor, hooking each piece separately to a machine that tests that piece alone. The end result is that you know exactly which piece is problematic and exactly where the work needs to be done.

    Boy, I wish my mechanic knew what unit testing means!!… 😛


Leave a Reply

Your email address will not be published. Required fields are marked *