I do web scraping on my daily basis. I was looking for the way to deal with the infinite scrolling using Selenium Python. One of the method I used while dealing with infinite scrolling data from chrome web store was to loop around the screen size.
number = 0
number2 = 2500
for i in range(105):
script_scroll = "window.scrollTo("+str(number)+","+str(number2)+")"
print(script_scroll)
contents = browser.find_elements_by_xpath("//div[@role='row']")
urls_list = ['urls']
for content in contents:
link = content.find_element_by_xpath(".//a")
link_url = link.get_attribute('href')
urls_list.append(link_url)
time.sleep(4)
browser.execute_script(script_scroll)
number = number2
number2 = number2 + number2
Top comments (0)