Screenshot Tests @ Top Eleven by Nordeus


#1

Purpose

Our goal with Screenshot Tests is to automate regression testing as much as possible. Before implementing Screenshot Tests we were spending ~25 hours on manual regression testing before each release. With Screenshot Tests we have reduced it to only around four hours.

We are using Screenshot Tests in two ways:

  • Client Screenshot Tests, with mocked data from server. These tests are pretty detailed and tend to cover all graphical components in Top Eleven
  • Full Integration Tests, in which client connects to test server. Only 10 - 15 smoke tests are executed
    This article will focus on Client Screenshot Tests - detailed testing of client side with mocked data from server.

Besides automating regression testing, Screenshot Tests bring some very positive side effects:

  • They are quick and easy way (especially for UI/UX designer) to check all graphical changes on client after new functionality is implemented and check whether client looks better than before
  • Easy way for e.g. Art Director or UI/UX designer to track evolution of certain screen in a new game
  • If developer needs to check some functionality that is hard to access / fake, dev can do it easily by just running appropriate test - no need for making bunch of code for faking data

How it works?

For Client Screenshot Tests, we are using mocked data from server. We have Unity Editor tool for recording responses from server. When test is run, client doesn’t connect to real server, but uses mocked data instead.

For Full Integration Tests, when test is run, test server is automatically deployed and client connects to it. Our data on server is different every time, so we execute only very basic smoke tests. For example, players in our Squad will be different every time we run the test. That’s why we don’t compare whole Squad screen, but only small part, making sure that screen is loaded and no errors occured. We do not dig into any particular data on the screen.

In our client code we have created functions that simulate users’ interactions with our client - clicking and dragging. Let’s say we want to test Finances screen in Top Eleven - we will write test that will click on Main Navigation, then click Finances button. After Finances screen is loaded it will click on a certain sponsor, click button for signing sponsor and so on. At several points during the test, screenshot is taken and sent to our simple PHP server for comparing images. Server compares the taken screenshot with the reference image. If any significant difference is noticed, test fails and gets reported to us.

So whatever gets changed on our client, we get informed. If change is intentional, we set new image as the reference image. If change occurred due to a bug, we fix the bug.

How do Screenshot Tests help us?

  • Finding various GUI bugs related to both rendering and interactions between GUI elements
  • Quick and easy way (especially for UI/UX designer) to check all graphical changes on client after new functionality is implemented and check whether client looks better than before
  • Easy way for e.g. Art Director or UI/UX designer to track evolution of certain screen in a new game

Coverage on Top Eleven

  • Screenshot Tests on Top Eleven are testing every part of Top Eleven client, performing basic actions
  • Tests are running on 4 real mobile devices
  • In each run ~550 screenshots are taken on each device, making it more than 2200 screenshots total per run (smile)

Some bugs found on Top Eleven by Screenshot Tests so far

  • Breaking bug in Tutorial - user unable to drag player because Mourinho is hiding it:

  • All opponent’s players in Squad having nine stars (smile) First image is reference image - how Squad should look like, second is how it actually looked due to bug. You can notice that opponent’s players (the ones displayed like shadows) have nine stars. On a third image differences are marked with red color:

  • Screenshot Tests are not only useful for finding technical bugs like these. They are also very useful tool for checking what changed on our client after something is implemented. Let’s take a look at how one UI design issue was found using Screenshot Tests. On first image we can see Player Popup before new Formations Improvements. On second image we can see how it looked like after Formations Improvements. On third image we can see differences marked with red pixels:

Looking at these images we could notice, for example, that instead of label “Foot: Right” now there is nice image of feet with right foot colored with green. This change is intentional and looks good. However we have also noticed that green arrow icon is changed and now doesn’t really look like an arrow on mobile devices:

It turned out that whole bunch of new icons were used by mistake, and our UI designer was super happy to see this fixed (smile)

Future

Our Screenshot Tests system is still not fully stable, and random crashes still happen, or execution sometimes stops due to various reasons. First we need to make sure all tests are completely stable - executing and passing.

After that we will implement functionality that when test fails, mail is sent to all developers who committed since that test last time passed. This way we will further shift mindset of developers towards fully maintaining Automation.