A
A
Andrey Ovchinnikov2018-08-28 22:46:36
Python
Andrey Ovchinnikov, 2018-08-28 22:46:36

Parsing a website using Selenium(PhantomJS)?

Shalom everyone. The thing is, I decided to parse links to Live matches from the site https://gg11.bet/ru/betting/?matchStatuses=LIVE. They suggested that Selenium (PhantomJS) can be used to work in the background. But the problem is this: I take a screenshot of the browser, and it shows that it does not execute Java scripts, and because of this, links cannot be parsed. Tried to change User-Agent - does not help. Below is attached class with bot

class Bot():  #Создаем класс бота
  def __init__(self):
    self.driver = webdriver.PhantomJS(desired_capabilities=dcap, executable_path=dictionary.PHANTOMJS_DIRECTORY) #Создаем драйвер браузера
    self.driver.set_window_size(1024, 768)
    self.navigate()

  def take_screenshot(self):
    self.driver.save_screenshot('сайт.png')

  def navigate(self): #Главная функция по автоматизации бота
    try:
      self.driver.get(dictionary.URL) #Здесь урл
      sleep(5)
      links = self.driver.find_elements_by_xpath(dictionary._XPATH) #Здесь он находит все ссылки
    except:
      self.driver.close() #Если вылезла ошибка - закрывает браузер
    else:
      for link in links: #печатает все ссылки(Links - список)
        print(link.get_attribute("href"))

I only know that I know nothing.©Socrates

Answer the question

In order to leave comments, you need to log in

1 answer(s)
A
Ambrosian, 2018-08-28
@Ambrosian

They suggested that Selenium (PhantomJS) can be used to work in the background.

Selenium is cumbersome and redundant and completely pointless. Only resources are consumed.
You need SlimerJS or PhantomJS
And yes - it is programmed in JavaScript
PS:
JavaScript is written, not a Java script.
Java script is different.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question