How to wait for YouTube selenium + phantom.js (c#) to load?

Question

YouTube

I'm trying to parse a list of youtube videos, accessing the page using Selenium + Phantom.js

IWebDriver webDriver = new PhantomJSDriver();
            webDriver.Navigate().GoToUrl(@"https://www.youtube.com");
            try
            {
                WebDriverWait wait = new WebDriverWait(webDriver, TimeSpan.FromSeconds(30));
                IWebElement firstResult = wait.Until(e => e.FindElement(By.CssSelector("ytd-rich-item-renderer")));
                Console.WriteLine(firstResult.Text);
            } catch(Exception e)
            {
                Console.WriteLine(e);
                Console.WriteLine(webDriver.PageSource);
            }
            
            Console.ReadLine();

In the code, I expect the page to load and the ytd-rich-item-renderer element to appear, but the element is not found

AND the page source code is displayed to the console before all scripts are executed, what could be the problem? Why didn't the page fully load in 30 seconds?



The screenshot shows that the page has not yet loaded and there is no necessary element.

I also inserted the received source code, which was displayed in the console in the code editor, and also did not find the necessary element, the page 100% does not have time to execute the scripts, but it is not clear for what reason

Answer 1

You can find everything from screenshots. Your search element is dynamic and is loaded separately. Moreover, you do not have a check for the number of elements.

using OpenQA.Selenium;
using OpenQA.Selenium.Firefox;
using OpenQA.Selenium.Support.UI;
....
....
....
using (IWebDriver driver = new FirefoxDriver())
            {
                WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(30));
                driver.Navigate().GoToUrl("https://www.youtube.com");
                wait.Until(web => {
                    var result = driver.FindElements(By.CssSelector("ytd-rich-item-renderer"));
                    return result.Count > 70;
                });
                var result = driver.FindElements(By.CssSelector("ytd-rich-item-renderer"));
                foreach (var element in result)
                {
                    Console.WriteLine(element.Text);
                }
            }

I tried like this. When I scroll the browser, the elements are loaded and as soon as there are more than 70 of them, it immediately goes to the next block of code.