DEV Community

Cover image for Scrapping Google Playstore
Anuoluwapo Balogun
Anuoluwapo Balogun

Posted on

Scrapping Google Playstore

Purpose Of Project :

To analyze how each fin tech apps is being used and widely accepted by Nigerians, as well as understanding the impact fin tech companies has made on Nigerians finances and Nigeri's Tech Sector.

Project Process:

  • Web scraping data from playstore
  • Performing Exporatory Data Analysis On data
  • Visualizing data on Power BI and Tableau

Project Tools:

  • Notebook
  • Power BI
  • Tableau

Importing Necessary Libraries

import time
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
Enter fullscreen mode Exit fullscreen mode

Accessing PlayStore Using Selenium

driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get('https://play.google.com/store/search?q=fintech%20in%20nigeria&c=apps')
time.sleep(10)
Enter fullscreen mode Exit fullscreen mode

Getting All The Apps In The Required Page

SCROLL_PAUSE_TIME = 5

# Get scroll height
last_height = driver.execute_script("return document.body.scrollHeight")
time.sleep(SCROLL_PAUSE_TIME)

while True:
    # Scroll down to bottom
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    # Wait to load page
    time.sleep(SCROLL_PAUSE_TIME)

    # Calculate new scroll height and compare with last scroll height
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height
Enter fullscreen mode Exit fullscreen mode

Scraping App Links

links_fintech = []
elems = driver.find_elements_by_xpath("//a[@href]")
for elem in elems:
    if "details?id" in elem.get_attribute("href"):
        links_fintech.append((elem.get_attribute("href")))

links_fintech = list(dict.fromkeys(links_fintech))
Enter fullscreen mode Exit fullscreen mode

Scraping The Necessary Informations From Each App

list_all_elements = []
for iteration in links_fintech:
    try:
        driver.get(iteration)
        print(iteration)
        time.sleep(3)

        header1 = driver.find_element_by_tag_name("h1")

        downloads = driver.find_elements_by_class_name("htlgb")
        list_downloads = []
        for x in range (len(downloads)):
            if x % 2 == 0:
                list_downloads.append(downloads[x].text)


        titles = driver.find_elements_by_class_name("AHFaub")
        comments = driver.find_element_by_class_name("EymY4b")

        list_elements = [iteration,header1.text, list_downloads.append(downloads[x].text), comments.text.split()[0]]
        for x in range (len(titles)):
            if titles[x].text == "Download":
                list_elements.append(list_others[x])
            if titles[x].text == "Developer":
                for y in list_others[x].split("\n"):
                    if "@" in y:
                        list_elements.append(y)
                        break

        list_all_elements.append(list_elements)
    except Exception as e:
        print(e)
Enter fullscreen mode Exit fullscreen mode

https://play.google.com/store/apps/details?id=ng.gov.cbn.speed.wallet.temp
https://play.google.com/store/apps/details?id=com.oxloan.loan.money.credit.nigeria
https://play.google.com/store/apps/details?id=com.cowrywise.android
https://play.google.com/store/apps/details?id=com.fastmoney.loan.credit.cash.nigeria
https://play.google.com/store/apps/details?id=com.wemabank.alat.prod
https://play.google.com/store/apps/details?id=com.ikonik.lamp_pay......

Creating A CSV File For Scraped DataFrame

import pandas as pd

df = pd.DataFrame(list_all_elements,columns=['URL', 'Name', 'downloads', 'install']) 
df_1 = df.to_csv('fintech_playstore.csv', header = True, index=False, encoding="utf-8")
Enter fullscreen mode Exit fullscreen mode
df_1
Enter fullscreen mode Exit fullscreen mode

https://www.danielherediamejias.com/scraping-google-play-store-with-python-and-selenium/

https://play.google.com/store/search?q=fintechs%20in%20nigeria&c=apps

Top comments (0)