Why doesn't puppeteer js wait for the selector that is displayed in the browser when parsing?

Question

Parsing

Good afternoon, there is a small parser on puppeteer.

Perhaps it is important that pages with the jsp extension.
Maybe the loading of dynamic content works in a special way than everywhere else ?!

Steps:

1) Authorization (works out)
2) Transition to the landing page. (works out)
3) Click on the loaded element. (problem) . When the parser browser is launched, the element appears after a negligible amount of time. Then it disappears again and reappears. (I don’t know how to deal with this)

None of the following tricks can save you as an expectation. All options were placed before the selector for the next click. None of the options helped:

1) await page.waitForNavigation();
2) await page.waitForSelector
3)
const sleep = (ms) => new Promise((res) => {
setTimeout(res, ms);
});
sleep(3000)

Код:

const puppeteer = require('puppeteer');

const config = require('./config'); // login, password

const sleep = (ms) => new Promise((res) => {
    setTimeout(res, ms);
});

const loginUrl = 'loginUrl';
const craftUrl = 'craftUrl';

(async () => {
    const browser = await puppeteer.launch({
        headless: false,
        args: ['--proxy-server=socks5://127.0.0.1:9050']
    });
    const page = await browser.newPage();

    await page.goto(loginUrl);

    await page.$eval('#usernameField', (elem, login) => {
        elem.value = login;
    }, config.login);

    await page.$eval('#passwordField', (elem, password) => {
        elem.value = password;
    }, config.password);

    await page.click('#ButtonBox > .OraButton.left');
    await page.waitForNavigation();

    await page.goto(craftUrl);

    sleep(2000);

        await page.waitForSelector('.textdiv').then(() => {
           let elements = document.getElementsByClassName('textdiv');
           elements[0].click()
        });
    

    sleep(5000);
    await browser.close();
})();

Выходит ошибка:

PAGE LOG: Window.onshow:[object PageTransitionEvent] trusted:true persisted:false
PAGE LOG: AppsLoginPage.loaded=true
PAGE LOG: call AuthenticateUser
PAGE LOG: Refused to set unsafe header "Connection"
PAGE LOG: Failed to load resource: the server responded with a status of 404 (Not Found)
PAGE LOG: window.unload:
PAGE LOG: [email protected]
steps.step1
steps.step2
^[[D

(node:5120) UnhandledPromiseRejectionWarning: TimeoutError: waiting for selector ".textdiv" failed: timeout 30000ms exceeded
at new WaitTask (/var/www/Parsers/teset/node_modules/puppeteer/lib/DOMWorld.js:549:28)
at DOMWorld._waitForSelectorOrXPath (/var/www/Parsers/test/node_modules/puppeteer/lib/DOMWorld.js:478:22)
at DOMWorld.waitForSelector (/var/www/Parsers/test/node_modules/puppeteer/lib/DOMWorld.js:432:17)
at Frame.waitForSelector (/var/www/Parsers/test/node_modules/puppeteer/lib/FrameManager.js:627:47)
at Frame. (/var/www/Parsers/test/node_modules/puppeteer/lib/helper.js:112:23)
at Page.waitForSelector (/var/www/Parsers/test/node_modules/puppeteer/lib/Page.js:1125:29)
at /var/www/Parsers/test/index.js:69:20
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:5120) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:5120) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Подскажите, в чем может быть причина?

Answer 1

await page.waitForSelector('.textdivresp').then(() => {
        console.log('textdivresp_ololo');
    });

    await page.evaluate(() => {
        let elements = document.getElementsByClassName('textdivresp');
        elements[1].click()
    });