I have recently watched a video from a Google conference called “The future of front end testing“, Presented by Greg Dennis & Simon Stewart.
Although this is a video done more than a year ago, I was amazed at how these exact issues are what I face in my day to day tests automation experience.
For those of you who intend to watch the video, the first 28 minutes should suffice. the rest are less relevant.
Prefer Unit Tests and Integration Tests over System Tests
I have previously addressed this issue in the “Why System Tests Can’t Replace Unit Tests” post.
System tests, naturally based on a lot of different components, are a lot more flaky.
They are the least precise tests – when they fail they give you as less info as possible.
Social Challenges VS Technical Challenges
I am a programmer, why do I need to write tests? I want to write features!
I have to admit that prior my work at sears israel, most of the places I worked in lacked this understanding.
developers wrote a set of minimal tests, if any, leaving the testing responsibility solely on the QA team shoulders.
a feature is not done until it is covered by automated tests. why?
let’s say you finished your feature and you manually tested it, deployed it to production and it all works fine. is that it?
what will happen if in another week someone else alters a related code, breaking your feature… how will that be detected?
Is that solely HIS responsibility? what if he had tests in your shared CI system that would indicate that he broke it?
Now think about YOU breaking someone else’s feature. wouldn’t YOU like to know it first?
The other issue is the ability of developer to understand the system structure, function and weak spots better than the QA team. YOU KNOW THE CODE! it’s as simple as that.
“I don’t know how to write tests for this”, “writing tests takes too much time”.
We all need to learn. all the time. that’s how it is.
Most companies have some internal testing frameworks\conventions. If writing a test is hard or requires too much time then you are probably doing something wrong. either because you don’t know how to or the framework is not written correctly.
I’ve never seen a system who had non-flaky system tests. we all suffer from that.
An untrusted test Is not a test at all… actually WORSE!
It casts on the entire suite and you’re better of just deleting it!
Correct Test Writing Patterns
Parallel Tests Execution
System tests are slow… as such, we should strive for parallel execution.
in order to be able to do that make sure to:
- Use a different account\user for each test, a different set of resources…
- Avoid shared resources or assign a Resource allocator that will lock these resource when needed.
Identify Non-Deterministic States
does your application rely on some configuration or external data and you find yourself altering your tests code from time to time to match these changes?
Add hooks to the application to allow you to address these changes. write to files\db and allow the testing code to access that data.
Wait for The Correct App State
Many tests rely on the fact that a particular state was reached.
a dialog opened\a certain page was reached. etc.
add validations to your code to make sure that the state was reached before continuing to the next steps.
DO NOT USE non conditional sleeps in your code. this introduces either flakiness or long running tests. both are bad!
Page Objects and IDs
There is a reason ids exist in pages. It allows you to identify you are where you think you are. use them!
The Page Objects model allows a separation of control of your app into the actual pages, each responsible for two things:
1) Identifying itself and it’s components (using ids).
2) Exposing the available functionalities.
Not Everything Should be Done Through the UI
E2E tests don’t need to be entirely through the UI.
certain parts of the system can be mocked-up, which is a lot cheaper, AS LONG AS the mocked functionality is tested elsewhere.
I have encountered several occasions of claims against this mocking approach, most of them are based on the notion the System Tests are there to test the System and as such, they should have no mock-ups, ore only for external parts.
If this functionality is tested elsewhere and there is a specific test that tests this functionality, why do I need to test it again and pay the runtime price of this element again?
there is a THIN line between making system tests behave as UI unit tests which should be addressed so use mock-ups wisely.
If needed, address the application DB directly to assert that what you expected really happened.
Tests Should be Able to Run on Different Environments
Other than your local system and you CI system, you might want to execute your tests on other environments (Development\Integration…).
avoid hard-coded environment specific values, like URLs.
Use Guice for different implementations.
Shared Resources and Tests Data
Each test needs data, whether created data or existing one.
your testing framework should support data injection or data retrieval according the test needs.
(of course, don’t forget to cleanup after yourself).
Better testing to all of us 🙂