I
I
Ivan2021-12-02 15:29:55
selenium
Ivan, 2021-12-02 15:29:55

Selenium performance issues in python?

There is a working project on selenium with ChromeDriver.
Faced a performance issue when scaling.
With 20 or more browser instances, the Load average rises to 15-20.
These settings corrected the situation a bit,

class Browser(validator.CategoryManager):
  def __init__(self, user_id, account_id):
    URL = ''
    CHROME_DRIVER_PATH = 'driver/chromedriver'

    options = ChromeOptions()
    options.add_argument('--headless')
    options.add_argument('--no-sandbox')
    options.add_argument('--disable-dev-shm-usage')
    options.add_argument('--disable-gpu')
    options.add_argument('--disable-application-cache')
    options.add_argument('--ignore-certificate-errors')
    options.add_argument('--blink-settings=imagesEnabled=false')
    #options.add_argument('user-agent={0}'.format('MQQBrowser/26 Mozilla/5.0 (Linux; U; Android 2.3.7; zh-cn; MB200 Build/GRJ22; CyanogenMod-7) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1'))
    options.add_experimental_option("excludeSwitches", ["enable-automation", "enable-logging"])
    options.add_experimental_option('useAutomationExtension', False)
    options.add_argument("--disable-blink-features=AutomationControlled")


The account is loaded by loading a cookie (To save space), then the script with validation is spinning, which performs N-th work on the site.
I do not know what can be disabled to improve performance.
I would not want to buy more hardware, I use yandex cloud, 24gb ram, 6vCPU

Answer the question

In order to leave comments, you need to log in

2 answer(s)
R
rPman, 2021-12-02
@rPman

Try changing the 'driver' from chromium to firefox
, it's not a fact that it will be faster, maybe even slower, but why not check?

A
ab1, 2021-12-21
@ab1

1. driver.close() driver.quit() is there?
2. parallelize on different machines - selenum grid
or run in the cloud in docker.
3. if the site is not difficult to parse with curl-om or via request libs)

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question