E
E
EduardMalinochka2020-12-23 16:34:07
Python
EduardMalinochka, 2020-12-23 16:34:07

Python selenium webdriver is not finding all elements on the page. How to fix it?

I need the driver to click on all the elements on the page and parse the information displayed on the click. To do this, I found elements by XPath and saved them to a sheet in order to write for loop to it. Everything worked as it should when there were a small number of elements on the page - 22. As soon as there are more elements, only a part of them are saved in my sheet and, accordingly, more than half of the elements are not clickable. The code runs and does not throw any error. How to fix it?

# login function
def login(email,password):
    driver.get('https://crm.tender-win.ru/account/logon')
    driver.find_element_by_id('email').send_keys(email)
    driver.find_element_by_id('password').send_keys(password)
    driver.find_element_by_id('btnLogin').click()

df = []
login('somelogin','somepassword')
driver.get('https://crm.tender-win.ru/tenders/new')
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, 'MyTenderMenuTabsInfo')))
tenders = [driver.find_elements_by_xpath("//div[@class='panel card ']") 
           or driver.find_elements_by_xpath("//div[@class='panel card new']")][0]

for tender in tenders:
    tender.click()
    WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, 'tenderCurrencyCode')))
    data = driver \
        .find_element_by_xpath('//*[@id="infoScroll"]/div/div[3]/div[2]/table') \
        .get_attribute('outerHTML')
    
    df1 = pd.read_html(data)
    df.append(df1)
    print(df)

Answer the question

In order to leave comments, you need to log in

2 answer(s)
V
Vladimir Kuts, 2020-12-23
@EduardMalinochka

Scroll the page and click again?

P
Pavel, 2020-12-24
@prafair

1) Remove [0] from the variable tenders
2) See what elements are saved in tenders. If not, then you need to wait a little longer. The element MyTenderMenuTabsInfomay be present, but not the fact that the rest of the elements had time to load, so check.
3) In a loop, first scroll to the element, click and check that the information specific to it has appeared, otherwise old information may be taken with a fast click and a slow browser

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question