What are flaky selenium tests? What are the reasons and symptoms of it? How can we try and minimise it? In this article, we are going to discuss these aspects in details.
What is flaky selenium test
A Flaky selenium test is a test which randomly fails even when there is no change in the code. Such flakiness gives the testers no confidence about the code as they are unaware of the automation script failure. Hence, the flakiness not only waste’s the time of the testers for re-running the script but also frustrates them as they have to debug the script for potentially no bug.
So what causes Flakiness?
Asynchronous calls: Suppose you are testing an application which gives asynchronous calls to other frames / forms then you have to impose a delay accordingly to wait for the call to complete successfully. Such kind of delay can be more or less than the time you have imposed in your automation script. This can increases the chances of script failure as you cannot be sure of the delay time.
Infrastructure challenges: These are external challenges due to which your test fails. Flakiness can be caused due to network slowness, DB slowness etc. The web driver may not support the jar files version, browser version is not compatible or the testing framework itself has some bugs, such issues might take a lot of time of the tester to identify the cause of the failure.
Integration with 3rd party tools and applications: You test your application to verify its integration with other components, applications or 3rd party tools. There are high possibilities of finding valid integration bugs while testing. But the chances of flakiness also increases since the external environments are not in our control. You may not run your test with external stub as they might not be available. Hence you have to test with 3rd party tools.
So now let’s check what kind of error appears on when test fails due to flakiness?
ElementNotFoundException: Even though the WebElement is present on UI, the error might appear due to flakiness as the webdriver couldn’t find the element.
ElementNotVisibleException: Your test might fail due to flakiness even though the WebElement is not hidden and visible on UI.
TimeoutException: The error might appear when the webdriver waits for a WebElement to be visible but due to flakiness the element doesn’t show up within the maximum webdriver waiting time.
InvalidElementStateException: You may get this error when the WebElement you are trying to interact is not in a state to perform the action which you would like. This occurs due to flakiness as the element might be obscured by another on clicking.
So, how to handle flakiness in selenium tests?
Flaky test is something which you cannot avoid. As your automation suit increases so does the chances of flakiness. Hence it is necessary to be preventive in your approach to handle flakiness rather than reactive.
One of the negative aspects of having too much flakiness is that it may start getting ignored even if it might be a genuine test failure of script defect.
So, here is a list of things we can do to prevent or to minimise the impact.
Develop a framework for your project: Designing the right test framework for your project is very important as it would relatively reduce the chances of flakiness. This eventually eradicates re-running of scripts and debugging them for flakiness issues. Your framework should be scalable and maintainable by you and your team.
Document Flakiness: You would have tried all possible ways to avoid flakiness but automation scripts may still fail. In order to save your future time and efforts you should document the finding on flaky tests. You can record your flaky test, gather logs, screenshots etc. which would help to tackle such issues in future.
A good document will make sure that the team is aware of the problems that caused the flakiness and has the resources to go back and fix it should the problem arise again.
Check your test environment: Unstable test environment may create flaky tests. A stable test setup without installation, browser or cache issues go a long way. In Fix Your Unstable Automated UI Tests blog post, Emanuil Slavov provides a first hand experiment of fixing test environment, to achieve 100% pass rate as compared to 50% because of flakiness.
We cannot eliminate flakiness in selenium but the impact and adverse effect of flakiness can definitely be minimised using measures suggested above. As an automation test engineer, we should be aware of these practical issues to become better and more efficient in our job.
Techcanvass is an IIBA endorsed education provider (EEP), iSQI ATP (for Certified Agile Business Analyst Training) as well as Agile Testing alliance partner for CP-SAT certification training in Selenium.
Techcanvass offers wide range of automation testing courses. These courses include Java and Selenium certification course, Python Selenium Certification course, BDD Training with Cucumber, RPA Training with UiPath, DevOps certification course and more automation testing courses.