There are generally 3 questions I approach automation with:
- What should be covered?
- How should it be covered?
- When should it be covered?
They are fairly simple questions, but they come with quite a bit of baggage.
What should be covered?
This is straightforward, it’s all about considering what should be covered with tests. There is a difference between what can and what should be covered. One could generally want to have everything covered with some form of automation, but I find a more important question to be what should be covered.
If there’s a number input field, you likely won’t be writing a billion tests that check for 1, 2, 3, etc. You will most likely cover boundary values, maybe input of non-integers, etc. If you have the same input field present in two places in the product - does it still make sense to cover that? Depending on what the functionality is, the answer could go both ways, even if that input field is the exact same one from a code perspective.
I have not come up with any set or rules for myself regarding this, but I still find it important to sit down and think (as well as discuss with other team members) what makes sense to be covered, and what doesn’t.
How should it be covered?
I come at this questions from different angles. I consider first where the coverage should take place - if it makes sense for it to be UI tests, integration tests, unit tests, etc. Below is an image of the testing pyramid (which I know one can consider as inaccurate as it’s very simple and does not encompass all testing levels - but its main point still stands).
As we can see, we should always strive to push tests to the lowest level possible. They are way cheaper and faster there, so this should be our goal.
The other portion of the question I refer to the testing data. When we consider how something should be covered, we need to be careful about the environment and data we’re using for the test. On the one hand we must ensure that we are using the right data for our tests, otherwise they will be of little use. In fact they can be damaging, if we’re testing for the wrong thing.
On the other hand, there can be ways to modify the environment to lower the amount of steps, making your tests faster, while still maintaining the same efficiency. If possible, then that’s the way to go!
When should it be covered?
A simple question, and it’s quite easy to answer with as soon as possible. That’s of course the goal one aims for. But the changes could be anything, from UI to huge code refactoring, and could be split into multiple stories so you may require a few of them to be in place before you can start doing actual tests.
Still, in light of that as soon as possible ideal, you could always prepare and plan the tests in advance, even when you cannot yet write them.