Why Selenium Tests Fail

Jason Huggins built an innovative test automation solution, referenced it as an antidote for mercury poisoning in chemical terms – Selenium.
For more than a decade, Selenium has revolutionized web application testing, enabling testing teams to automate tests that could earlier be run only manually.

Theoretically, it has freed up QA Engineers to focus on more crucial tasks rather than doing repetitive regression tests manually. Practically, anyone who has used Selenium extensively will know that the majority of your time goes on troubleshooting tests, dealing with failing tests, test maintenance and dealing with flaky tests.

Below I have listed out various reasons for failures in selenium tests:

1. Web Element positioning changes on the screen
   
   The most common cause of selenium failures is attributed to the change in the web element position on the screen. This has increased with the growth of responsive applications where the screen layout dynamically changes with the screen size. Even the minor CSS styling changes, can end up with tests failing to select the correct element.

2. Unexpected Window-based pop-ups
   
   Another reason for a test failure in selenium is when there occurs an unexpected window based pop up while the test case is being executed as selenium is capable of handling on the web-based pop-ups and would require a third-party tool integration to achieve success in this case.

3. Over-precise comparisons
   
   Comparisons are used to check whether an action has been completed properly. For instance, you might check that the total value of a shopping cart is correct. If you make your comparison too precise you run the risk that a minor change may make the comparison fail. For instance, if you check for an exact sum for the shopping cart, then any price change in your database will cause this comparison to fail.

4. Dynamically changing data-sets
   
   Selenium test is often run with a known dataset. Changes to the data can trigger all sorts of test issues, which can be hard to zero down. This problem occurs because often you want to test your system with real data which may be dynamic by default.

5. Unknown state of the system
   
   With the dynamic nature of modern web application, the current state of the system at any given instance can have a huge impact on testing. By the state of the system, it refers to the state of system logs that have exceeded the defined size and are slowing things down, or the state of the system libraries being called, updated or replaced. Modern libraries such as React or Angular.js adds to this problem because they rely on client-side data models and dynamic element generation in response to user actions.

6. Errors discovered later than occurred
   
   Sometimes with web application testing, a test step may be incorrect, but the action will succeed and the test will move ahead without any issue. Then, within the test, this downside can come about and trigger a failure. The problem is that the foundation explanation for this failure was several steps earlier.
   
   Imagine the checkout flow for an e-commerce app. After filling in their email address, a user can click to create an account and proceed to checkout. On redesigning the page, the “Create Account” button is now moved to a different part of the page and is renamed as “Register”.
   In the original place of Create Account button, there is now a new “Checkout as Guest” button.
   
   The test can still blithely proceed all over checkout. But if the final step is to check the order status in the user account page the test will now fail.

7. Timing, not enough or timing too long
   
   Selenium tests include performing a set of actions in a specific sequence. In case of a change in the sequence or if any one of the actions is delayed, the test is bound to fail.
   
   This can be explained in more detail in the login test case example below with a sequence of actions:
   
   1) Click to open the login screen;
   2) Enter username;
   3) Enter the password;
   4) Click login;
   5) Check that you are now able to access your profile from the
   homepage
   
   In the above example, items 2 and 3 can be swapped, but not in the case when the password box is to be displayed only after the username has been entered. However, all the other actions have to happen in the correct order and can only happen once the previous action has been completed. If the system is under heavier load or in case of a weak network connection, while running the test, there may be a delay between clicking login and going to the homepage. In this case, the test will fail.

8. Dynamic pages
   
   Fully dynamic sites pose major issues for Selenium and. In a dynamic site, the HTML is generated on-demand by scripts running in the background and is then displayed on the screen. Content getting displayed on the web page is determined by several factors which keep changing dynamically.
   
   For instance, a page may display random suggestions to buy and sell stocks based on the share market sentiments. This sort of dynamic behaviour means that Selenium will either never be able to find the correct selector, or may break in future comparisons even if found.
   
   Example: Suppose there is a SUBMIT button on the page whose id is X123 and there is a drop-down selection before that on changing of that drop down this submit button DOM element Id changed like sometimes it shows X124 or X125 etc. then in selenium script we only maintained X123 so it should be a problem and script failed.

9. iFrames
   
   Many Web applications use iFrames to embed content. For instance, they may embed various ads in iFrames. Sometimes these iFrames may even be nested (iFrames in iFrames). Dealing with these iFrames requires significant coding effort.
   
   The iframes are separate documents embedded in DOM, and their contents will not be included in the HTML code of the main page; iFrames have to be read separately.
   
   This is because the system has to keep track of Document Object Model where the iFrame belongs to before selecting an element and may require the tester to modify the script to switch between DOMs during the test flow.

With Selenium you can automate web applications on different browsers, however, Selenium test scripts are difficult to maintain with objects and attributes and are not always reliable. Hence it isn’t solely reliable.
You may find more Selenium alternatives that make the test automation process easy and help achieve continuous testing and make less flaky and easy steps.

Further Reading:
The Good and the Bad of Selenium Test Automation Tool
Pros and Cons of Selenium as an Automation Testing tool