Tag Archives: System tests

Front-End Testing Done Right

I have recently watched a video from a Google conference called “The future of front end testing“, Presented by Greg Dennis & Simon Stewart.

Although this is a video done more than a year ago, I was amazed at how these exact issues are what I face in my day to day tests automation experience.

For those of you who intend to watch the video, the first 28 minutes should suffice. the rest are less relevant.

 

Prefer Unit Tests and Integration Tests over System Tests

 

I have previously addressed this issue in the “Why System Tests Can’t Replace Unit Tests” post.

System tests, naturally based on a lot of different components, are a lot more flaky.

They are the least precise tests – when they fail they give you as less info as possible.

 

 

Social Challenges VS Technical Challenges

 

Social Challenges

I am a programmer, why do I need to write tests? I want to write features!

sounds familiar?

I have to admit that prior my work at sears israel, most of the places I worked in lacked this understanding.

developers wrote a set of minimal tests, if any, leaving the testing responsibility solely on the QA team shoulders.

a feature is not done until it is covered by automated tests. why?

let’s say you finished your feature and you manually tested it, deployed it to production and it all works fine. is that it?

what will happen if in another week someone else alters a related code, breaking your feature… how will that be detected?

Is that solely HIS responsibility? what if he had tests in your shared CI system that would indicate that he broke it?

Now think about YOU breaking someone else’s feature. wouldn’t YOU like to know it first?

The other issue is the ability of developer to understand the system structure, function and weak spots better than the QA team. YOU KNOW THE CODE! it’s as simple as that.

 

Technical Challenges

“I don’t know how to write tests for this”, “writing tests takes too much time”.

We all need  to learn. all the time. that’s how it is.

Most companies have some internal testing frameworks\conventions. If writing a test is hard or requires too much time then you are probably doing something wrong. either because you don’t know how to or the framework is not written correctly.

 

Flaky Tests

 

I’ve never seen a system who had non-flaky system tests. we all suffer from that.

An untrusted test Is not a test at all… actually WORSE!

It casts on the entire suite and you’re better of just deleting it!

 

Correct Test Writing Patterns

 

Parallel Tests Execution

System tests are slow… as such, we should strive for parallel execution.

in order to be able to do that make sure to:

  • Use a different account\user for each test, a different set of resources…
  • Avoid shared resources or assign a Resource allocator that will lock these resource when needed.

Identify Non-Deterministic States

does your application rely on some configuration or external data and you find yourself altering your tests code from time to time to match these changes?

Add hooks to the application to allow you to address these changes. write to files\db and allow the testing code to access that data.

 

Wait for The Correct App State

Many tests rely on the fact that a particular state was reached.

a dialog opened\a certain page was reached. etc.

add validations to your code to make sure that the state was reached before continuing to the next steps.

DO NOT USE non conditional sleeps in your code. this introduces either flakiness or long running tests. both are bad!

 

Page Objects and IDs

There is a reason ids exist in pages. It allows you to identify you are where you think you are. use them!

The Page Objects model allows a separation of control of your app into the actual pages, each responsible for two things:

1) Identifying itself and it’s components (using ids).

2) Exposing the available functionalities.

 

Not Everything Should be Done Through the UI

E2E tests don’t need to be entirely through the UI.

certain parts of the system can be mocked-up, which is a lot cheaper, AS LONG AS the mocked functionality is tested elsewhere.

I have encountered several occasions of claims against this mocking approach, most of them are based on the notion the System Tests are there to test the System and as such, they should have no mock-ups, ore only for external parts.

If this functionality is tested elsewhere and there is a specific test that tests this functionality, why do I need to test it again and pay the runtime price of this element again?

there is a THIN line between making system tests behave as UI unit tests which should be addressed so use mock-ups wisely.

If needed, address the application DB directly to assert that what you expected really happened.

 

Tests Should be Able to Run on Different Environments

Other than your local system and you CI system, you might want to execute your tests on other environments (Development\Integration…).

avoid hard-coded environment specific values, like URLs.

Use Guice for different implementations.

 

Shared Resources and Tests Data

Each test needs data, whether created data or existing one.

your testing framework should support data injection or data retrieval according the test needs.

(of course, don’t forget to cleanup after yourself).

 

 

Better testing to all of us 🙂

 

 

 

Not Everything Can Be Automated! Really!

Following my previous post, regarding System Tests misconception, I would like to address another misconception.

NOT ALL TESTINGS CAN AND SHOULD BE AUTOMATED!

Some of you might think it is obvious. Well, good for you!

but… it’s amazing how many people miss this point.

I have been developing and writing automated tests and platforms for the past 5 years.

 

In the beginning…

 

When I first started developing automated tests for several companies I was amazed how few companies had actual automated tests.

Most had only manual testers, Some had semi manual tests, where the data creation was automated and the analysis was manual.

Only a few had actual Automated tests and a fewer knew how to actually write and maintain them correctly.

The notion of automated tests, at that time, was not easily grasped. There were claims that it will take a lot of time to build a basic framework for the tests, that the maintenance is too expensive, that it wouldn’t be scalable enough…

It took some time, but now we are at the opposite concept, were a lot of people believe everything can and should be automated.

Of course Automation is great. No argument here.

It allows you to have continuous integration environments, It is not human dependent and it can be very scalable.

Still, it has it’s downfalls.

 

What Can be Automated?

 

Technically speaking. all tests that require non graphical interactions (mainly composed of API calls, data analysis, Simple controller testing) and have no actual GUI, can  be automated.

Those are mostly tests which don’t require any human interaction, are straightforward and well defined functional tests.

Their objective is to see everything FUNCTIONS as it should.

 

So, if it’s technical possible, then they should always be automated?

Here comes the ROI issue into consideration.

Yes, writing an Automated test will sometime require a lot more time than manually testing it.

  • Time needed for developing the technical framework to support the requested functionality.
  • Data bootstrap which is needed for the testing environment.
  • Tests that will require a lot of maintenance or could be unstable in some environments.

 

So… What can’t be automated (Or rather shouldn’t)? 

There are a few issue in which automation is still lucking the proper solution.

A) Quality:

There are a few solutions out there that suggest they have the ability to estimate the quality of your product.

Up until now, I have yet to encounter an actual framework that gives a proper comprehensive automated solution.

In the past, I have worked quite a lot scheduling schools timetables. you can read more about it in my Jobs Diversity post.

There was a semi automated system that tried to produce a well balanced and fit schedule for all the teachers, but… no matter what, It was always down to some human considerations to wrap it up properly.

You couldn’t give up the human sense when it came to the need to compromise.

Today, I work at Sears Israel, where we are developing a Social e-Commerce and have a great need to examine our recommendations quality.

We do have several ideas regarding how to try and define quality measurement so they can be ran automatically, but still, they are not human-less.

Another example is a Mobile Navigation company I have worked with in the past which had a quality automated system in which they initially took a snapshot of what they considered to be a “Good quality output” and thereon relied on that fact to make sure all future results resemble the initial one. that sounded promising at the time, but…

After a while, the base snapshots where less and less examined, which resulted in bad quality being verified to persist being bad all across the system 🙂

 

B) Visual Layout and UI:

GUI elements can be tested for functionality. sure.

You have Jemmy for Java Swing applications, Selenium for web automation and more…

BUT, they are all good mostly at testing functionality. You can fetch an element X,Y location with these tools but you will not be able to ensure it appears correctly.

You can semi automate it and take screen capture of the app and later on examine them, but as I said, that’s not a fully automated process.

So.. you still need to “have a look” and see nothing visual was broken.

Remember: a selenium test will pass even if your button is located completely out of place, just as long as it is located and visible.

 

C) User Experience:

This is maybe the most important of them all.

You covered the functionality, The quality seems good and you even semi automated your screen captures. is that enough to guarantee a quality product?

hmmm…. what about the actual human who is going to use it? is the overall experience of a user on your site good? are the actions self explanatory? are the messages visible enough?

is he overloaded with too much info? and many many more questions can be asked regarding his experience.

Most of these questions cannot be answered in the context of a single feature but rather as a combination of several brought together.

 

There are a few more issues, but these are the biggest ones in my perspective.

So… don’t give up on your need for manual testing yet. We have still a way to go to get there, if ever…

 

Why System tests can’t replace Unit tests

Several times I have come across claims of redundant testing.

Most of which claim that a System test can cover all the functionality, so why do we need a unit test?

First of all I would like to emphasize their differences.

Unit tests

Unit tests are basically short and specific.

Checking your helper class returns the right results, test your algorithm on different inputs and checking all edge cases. It is the smallest and basic testable part in an application.

They test that the code actually does what it should… CODE WISE.

Due to their nature, they usually don’t need any third parties or other external dependencies. they are self contained.

System Tests

A complete and integrated testing to make sure all the different parts of the system work together as expected.

These tests are usually more complicated. Many of them will include third parties for their automation, they fall into the “black box testing” criteria and they usually test the functionality of the application as a whole.

Wouldn’t a system test be able to cover any unit test?

Absolutely not!

Think about it.

Even if the same data is created in the beginning of the test and the same assertions are made, how can you be certain who is responsible for the success or failure? there are usually several components, some of which might affect the result.

The test can fail due to several components which will require more investigation, drilling into each components in any case, making sure “this part up until here” works OK and will eventually lead to more wasted time asking yourself “why didn’t I write a unit test for this”.

If the test passed, does it guarantee that everything is OK?

Your first intuition would be YES. but… you’d be wrong 🙂

This only means the SYSTEM is OK at this point. maybe your code has an error which is luckily missed due to another component, ran after your code.

What will happen if that piece of code changes and the actual bugs appear? how much time will you spend tracing down the bug?

Moreover, system tests will usually cover main flows, not edge cases.

We can write system tests for all edge cases.

Yes, you CAN. but how effective will they be? how long will they run?

System tests are almost always slower then unit tests. if not, something is wrong with your unit tests 🙂

Secondly, as explained before, that will most likely violate the “black box testing” methodology since you will have to be aware of the edge cases IN THE CODE.

Another issue is catching bad commits. executing tests before committing will detect your errors.

No problem here, All our programmers execute tests before they commit.

Do they now?

As per my experience, most developers will run the minimal set of tests they can to make sure they did not break anything. They will search for all related unit tests using references to the changed functions or using some simple type searching. most of the time they will not be even aware you have system tests that cover the part they’ve changed.

Still, if we make sure everyone runs ALL the tests and we do not suffer from long execution times, why do we still need unit tests?
that’s simple, because of the “ignored test” syndrome.

never heard of it? c’mmon…

when a test is unstable or fails and you have no time to fix it, you just put it in ignore or ignore it anyway.

but, since this is a system test it checks several components, meaning you didn’t just disable that specific test but all other “disguised” tests for sub-components. had you written unit tests, this test had less effect on the questions: “should we fix it now? is it top priority? is anything else covering this issue?”

To conclude,

System tests and Unit tests each have a specific role. They complement each other.

If you are pressured, System tests will cover basic functionality. but if you have the time, do both, ALWAYS!

For more formal definitions look at:

Unit Tests

System Tests

Unit tests vs System tests