Tag Archives: selenium

Not Everything Can Be Automated! Really!

Following my previous post, regarding System Tests misconception, I would like to address another misconception.


Some of you might think it is obvious. Well, good for you!

but… it’s amazing how many people miss this point.

I have been developing and writing automated tests and platforms for the past 5 years.


In the beginning…


When I first started developing automated tests for several companies I was amazed how few companies had actual automated tests.

Most had only manual testers, Some had semi manual tests, where the data creation was automated and the analysis was manual.

Only a few had actual Automated tests and a fewer knew how to actually write and maintain them correctly.

The notion of automated tests, at that time, was not easily grasped. There were claims that it will take a lot of time to build a basic framework for the tests, that the maintenance is too expensive, that it wouldn’t be scalable enough…

It took some time, but now we are at the opposite concept, were a lot of people believe everything can and should be automated.

Of course Automation is great. No argument here.

It allows you to have continuous integration environments, It is not human dependent and it can be very scalable.

Still, it has it’s downfalls.


What Can be Automated?


Technically speaking. all tests that require non graphical interactions (mainly composed of API calls, data analysis, Simple controller testing) and have no actual GUI, can  be automated.

Those are mostly tests which don’t require any human interaction, are straightforward and well defined functional tests.

Their objective is to see everything FUNCTIONS as it should.


So, if it’s technical possible, then they should always be automated?

Here comes the ROI issue into consideration.

Yes, writing an Automated test will sometime require a lot more time than manually testing it.

  • Time needed for developing the technical framework to support the requested functionality.
  • Data bootstrap which is needed for the testing environment.
  • Tests that will require a lot of maintenance or could be unstable in some environments.


So… What can’t be automated (Or rather shouldn’t)? 

There are a few issue in which automation is still lucking the proper solution.

A) Quality:

There are a few solutions out there that suggest they have the ability to estimate the quality of your product.

Up until now, I have yet to encounter an actual framework that gives a proper comprehensive automated solution.

In the past, I have worked quite a lot scheduling schools timetables. you can read more about it in my Jobs Diversity post.

There was a semi automated system that tried to produce a well balanced and fit schedule for all the teachers, but… no matter what, It was always down to some human considerations to wrap it up properly.

You couldn’t give up the human sense when it came to the need to compromise.

Today, I work at Sears Israel, where we are developing a Social e-Commerce and have a great need to examine our recommendations quality.

We do have several ideas regarding how to try and define quality measurement so they can be ran automatically, but still, they are not human-less.

Another example is a Mobile Navigation company I have worked with in the past which had a quality automated system in which they initially took a snapshot of what they considered to be a “Good quality output” and thereon relied on that fact to make sure all future results resemble the initial one. that sounded promising at the time, but…

After a while, the base snapshots where less and less examined, which resulted in bad quality being verified to persist being bad all across the system 🙂


B) Visual Layout and UI:

GUI elements can be tested for functionality. sure.

You have Jemmy for Java Swing applications, Selenium for web automation and more…

BUT, they are all good mostly at testing functionality. You can fetch an element X,Y location with these tools but you will not be able to ensure it appears correctly.

You can semi automate it and take screen capture of the app and later on examine them, but as I said, that’s not a fully automated process.

So.. you still need to “have a look” and see nothing visual was broken.

Remember: a selenium test will pass even if your button is located completely out of place, just as long as it is located and visible.


C) User Experience:

This is maybe the most important of them all.

You covered the functionality, The quality seems good and you even semi automated your screen captures. is that enough to guarantee a quality product?

hmmm…. what about the actual human who is going to use it? is the overall experience of a user on your site good? are the actions self explanatory? are the messages visible enough?

is he overloaded with too much info? and many many more questions can be asked regarding his experience.

Most of these questions cannot be answered in the context of a single feature but rather as a combination of several brought together.


There are a few more issues, but these are the biggest ones in my perspective.

So… don’t give up on your need for manual testing yet. We have still a way to go to get there, if ever…