I am writing a couple of codes to get the urls from yahoo homepage image slider window which is embedded through javascript, therefore I use Selenium to simulate a human click action. The code and details illustration is attached in the python file. Please help me to figure it out.
'''Since Ubuntu do not have physical FireFox Browser, so I use
xvfbwrapper to create the environment,
I only add it on the server site. Local testing, I did not add it.
'''
#from xvfbwrapper import Xvfb
#with Xvfb() as xvfb:
from selenium import webdriver
import selenium.webdriver.support.ui as ui
url_path="https://sg.yahoo.com"
driver = webdriver.Firefox()
driver.get(url_path)
#wait until the Crawler find the Carousel-Ranges Class which is the image slider (it contains all the urls which i need)
first_result = ui.WebDriverWait(driver, 10).until(lambda driver: driver.find_element_by_class_name('Carousel-Ranges'))
#get the number of url to implement the for loop
imageRange = first_result.text
numString = imageRange.split("of ")
numInt = int(numString[1])
print numInt #print total number of URL that extracted through text
for x in range(0,((numInt-10)/5)+1):
#simulate the click action. every single click, the javascript will enable 5 more urls so I can use "li.Cur-p a"
#to get, and by default there are only 10 urls which are enabled.
driver.find_element_by_css_selector("button[class*='End-0 T-0 B-0 Fz-30 Z-1']").click()
topNews = driver.find_elements_by_css_selector("li.Cur-p a");
#print out the urls list that cralwed. If it matches with previous number
# then it means I successfully get the result.
print len(topNews)
But this code is only workable for the local testing. When I add the xvfbwrapper and put it into the server, it does not work. Taking an example, if the total urls is 60, the crawler runs on local that returned the same result(60-success); but when server runs it, it only returns 10 which is default urls. Therefore, I deduce the problem may cause by this xvfbwrapper. But I do not how to solve it. Can anybody address on this matter? Thanks again
xvfbwrapper
?