Advanced Automation Tips with Python | Selenium

This post cross-published with OnePublish

What's up DEV Network?

In this post I want to share my automation experience from my current internship.

Actually, I wanted to explain it to you on a real project, but unfortunately, I didn't find any platform that fits all my requirements.

However, if you are Python developer you will need these tips to build your automation program fast. So, let's start!

Web Driver

It is important to configure web driver correctly to be able to run automation. If you want to use Chrome as a web driver then you should install chromedriver. However, if you want to choose Firefox then you should install geckodriver.

Creating a class

With creating a class you can easily handle URLs and call functions so you don't have to create web drivers in each function. Take a look following code:

from selenium import webdriver

class Bot:

    def __init__(self):
        self.bot = webdriver.Firefox(executable_path='/path/to/geckodriver')

automate = Bot() 

So, when you need driver in function just write:

  def search(self):
        bot = self.bot
        bot.get('www.google.com')    

Handle Login

For handle login you should add fields into __init__ function. Take a look following code:

import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

class Bot:

    def __init__(self,username,password):
        self.username = username
        self.password = password
        self.bot = webdriver.Firefox(executable_path='/path/to/geckodriver')

    def login(self):
        bot = self.bot
        bot.get('https://website.com/')
        time.sleep(3)
        email = bot.find_element_by_id('userNameBox')
        password = bot.find_element_by_id('passWordBox')
        email.clear()
        password.clear()
        email.send_keys(self.username)
        password.send_keys(self.password)
        bot.find_element_by_id('LoginBtn').click()
        time.sleep(3) 

automate = Bot('your_username', 'your_password') 
automate.login()

Very simple approach to handle logins.

Looping clicks

It is important to use time.sleep() if you are looping clicks. Because automation needs time to perform another click. If you don't use time.sleep() then ElementClickInterceptedException error will appear.

for elem in elements:
        elem.click()
        time.sleep(3)

Write data into CSV

There is a lot of solutions and examples on the internet about how to write data to CSV. Let's assume that you have multiple arrays and you want to write them into CSV under the right field names (column name).

Let me be more clear..

My task was to crawl some phrases and translate them to all languages then write all these data into CSV. So, I used GT API and created an array for each language.

I had a lot of arrays so I need somehow write each translation under right field names (column name).

The following code demonstrating the best solution to handle these kinds of problems:

# zip arrays
languages_translations = zip(ar,hy,ms,bg,zh_cn_SAR,zh_cn,
zh_tw_singapore,zh_tw,hr,cs,da,nl,en_australia,en_uk,en_usa,et,fi)    

# Write to CSV
with open('translations.csv', mode='w', newline='', encoding="utf-8") as csv_file:
    fieldnames = ['Arabic', 'Armenian (Armenia)', 'Bahasa Malaysia (Malaysia)', 'Bulgarian (Bulgaria)', 'Chinese (Hong Kong SAR)', 
    'Chinese (Simplified)','Chinese (Singapore)','Chinese (Traditional)','Croatian (Croatia)', 'Czech (Czech Republic)',
    'Danish (Denmark)','Dutch (The Netherlands)','English (Australia)','English (UK)', 'English (US)','Estonian (Estonia)','Finnish (Finland)']
    writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
    writer.writeheader()
    for lang in languages_translations:
        writer.writerow(dict(zip(fieldnames, lang)))  

As you see, you should zip all arrays, loop it and write rows dictionary of zipped field names and loop items.

By this way you will write all data correctly into CSV file.

Handle Popups or Iframes

Another challenging task is to handle popups or iframes while automation. If you need to interact with popup or iframe elements, you should tell selenium to switch the window from main to popup. Once you finished with popup you should switch back to the main window.

# find iframe or popup element
iframe = bot.find_element_by_tag_name('iframe')
# switch to iframe or popup window
bot.switch_to.frame(iframe)
# interact with elements
textarea = bot.find_element_by_tag_name('body')
textarea.send_keys('Some Keys Here')
# switch back to main window
bot.switch_to.default_content() 

Great! I am providing you the solutions which is really complicated on internet.

If you have to handle alert popups you also can use this method.

Send CSV data to web elements

Sometimes you need to send csv data into web elements such as multiple textboxes.

Assume that you have a multiple panels with multiple textboxes in it and you have to send CSV rows into these texboxes

So each CSV row belongs different panels. Sounds crazy right? This is how you will train your brain :) Somehow you have to iterate each CSV row for each panel's textboxes.

Before we go on please visit Reverse Python if you want to find more articles like this.

Alrgiht! Let's see the code and then I am going to explain:

def send_keys_textboxes(self,url):
    bot = self.bot
    # go to url
    bot.get(url)
    # reader object which will iterate over lines in the given csvfile
    with open('translations.csv', 'rt', encoding='utf8') as csvfile:
        langs = csv.reader(csvfile, delimiter=',', quotechar='"')
        # create list of langs
        langs = list(langs)
    elements = bot.find_elements_by_xpath("//a[@data-tag='globalize']")
    # Using index to handle multiple panels
    index = 0
    try:
        for elem in elements:
            class_of_element = elem.get_attribute("class")
            if class_of_element == 'cs-trans-icon':
                # Panel opens with click
                elem.click()
                time.sleep(3)
                textBoxes = bot.find_elements_by_tag_name('textarea')
                # Loop only specific index of list 
                phrases = langs[index]
                # Itearating TextBoxes
                for i in range(len(phrases)):
                    textBoxes[i].send_keys(phrases[i].title())
                time.sleep(3)
                # Increasing index for next panel
                index = index+1
                try:
                    bot.find_element_by_class_name('CsImageButton').click()
                except NoSuchElementException:
                    bot.find_element_by_class_name('cso-btn').click()
            time.sleep(3)  
    except ElementClickInterceptedException:
        pass 

Index plays main role in this solution. We said that we want to handle multiple panels, so index prevents to loop all rows in each panel. Assume that you have 3 rows and without using index, program will continue to loop until 3rd row finished. As a result you will have 3 values for each textbox. However, we need to send only first row values into first panel's textboxes, second row values into second panel's textboxes and so on..

Once index is defined, we are iterating textboxes. By this way, first value in row is going to send first textbox, second value in row is going to send second textbox and so on..

These examples are from the real-world project so I recommend to bookmark this post.

Handle Dropdowns

So there are 2 approaches for handling dropdwons.

First solution is to use selenium's Select method to select options from dropdown.

from selenium.webdriver.support.ui import Select

bot = self.bot
bot.get("https://example.com")
select = Select(bot.find_element_by_xpath("//select"))
# select bu index of option
select.select_by_index(2)
# select by text of option
select.select_by_visible_text('Visible Text')
# select by value of option
select.select_by_value('value')

and second solution is just using xpath to click dropdown items:

bot.find_element_by_xpath("//select/option[text()='Option_Text_Here']").click()

Navigate parent element with xpath

Sometimes Html structures can be messed up and you can't select the element you want. In these cases, it is good to navigate the parent element or go back using xpath:

# elem is a web element object
# selenium will jump 2 step back to look for parent element
 parent_element = elem.find_element_by_xpath('..').find_element_by_xpath('..')

Mission Accomplished!

That is all for now! I shared my python automation experience with you and I hope it helps you and saves your time from searching on the internet.

Please visit Reverse Python for more articles (you won't disappointed) and let me know in comments what my next post should be about.

See you very soon DEVs! Stay Connected!