How Selenium Works: Episode 4 - Finding Elements

After an interaction on the last weekend of January 2020, on a Selenium Issue where someone said “why can’t you just…” after I explained the issue I thought that I would start explaining commands in Selenium WebDriver and why we landed on the design that we have today.

I will repeat this on every page of the series but a lot, an annoying amount sometimes, of thinking goes into how every little bit of Selenium works.

In this episode we are going to look at how findElement works. We need to be able to find elements before we can interact with them.

Finding an element is easy right?

Well, yes if people designed their applications for testability over anything else! Unfortunately we live in a world where most of the time people think about quality and testing as an after thought.

Anyway... so let's look at the different ways we can look them up.

Search Types

This part is simple, Selenium offers the same methods for searching as you can find when interacting with the DOM with JavaScript. If we can look up things via document.querySelectorAll(aQuery); then it will be searchable via find_element or find_elements. This is searchable from the WebDriver object or from the WebElement object. If you do element.find_element(...) that is equalivent to doing element.querySelectorAll(aQuery) which sets the starting point, or the root, for searching from that element.

Below is an example of how to use find_element:

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Firefox()
driver.get("https://www.theautomatedtester.co.uk")
driver.find_element(By.ID, "someID")
driver.find_element(By.CSS_SELECTOR, "#someID")
driver.find_elements(By.CSS_SELECTOR, "#someID")
driver.find_element(By.NAME, "someName")
driver.find_elements(By.NAME, "someName")
driver.find_element(By.CLASS_NAME, "someName")
driver.find_elements(By.CLASS_NAME, "someName")
driver.find_element(By.TAG_NAME, "someName")
driver.find_elements(By.TAG_NAME, "someName")

We can also look up elements on the page by XPATH.

We can do the following

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Firefox()
driver.get("https://www.theautomatedtester.co.uk")
driver.find_element(By.XPATH, "//li")

We can also look up links based on the visible text. Visible text uses the }}">isDisplayed algorithm to work out what people can and can't read. Selenium will collect all the <a></a> tags on the page, get the visible text, and then find the first element that has that text.

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Firefox()
driver.get("https://www.theautomatedtester.co.uk")
driver.find_element(By.LINK_TEXT, "some text")
driver.find_element(By.PARTIAL_LINK_TEXT, "text")

What happens when it finds the element?

When we are able to find an element we keep track of it in a Map. The Map will contain a representation of a element using uuid. The map allows us to look up elements when we get them back from a Selenium test and make sure that element is actually connected to the DOM. If the element is not connected to the DOM and you try use it you will get a StaleElementReferenceException.

and when it doesn't find the element or elements

There are 2 things that can happen.

If you look for only 1 element you will get an exception like:

from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By

driver = webdriver.Firefox()
driver.get("https://www.theautomatedtester.co.uk")
try:
    driver.find_element(By.LINK_TEXT, "some text")
except NoSuchElementException:
    pass

If you are looking for more than 1 element you will get an empty list:

from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By

driver = webdriver.Firefox()
driver.get("https://www.theautomatedtester.co.uk")
elements = driver.find_elements(By.CSS_SELECTOR, "myShadowElements")

assert len(elements) == 0

Coming Soon: Relative locators

These allow you to search for elements on the page by their relative position to another. For example you can find the element that is above and to the left of another element. See below

from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.relative_locator import with_tag_name

driver = webdriver.Firefox()
driver.get("https://www.theautomatedtester.co.uk")
elements = driver.find_elements(
                    with_tag_name("td").above(
                    driver.find_element(By.ID, "center"))\
                    .to_right_of(driver.find_element(By.ID, "second")))

Further reading

You can read further in the Element Retrieval section of the WebDriver Specification