How to click the "Next" button on the site, and save the page with changes?

Question

Python

There is such a site: https://www.b17.ru.
It publishes posts about psychology, I wanted to parse headlines from it and send me a message to my desktop about new posts. I do this for practice in Python.
But here's the problem, not all new posts are displayed on the main page, but only the most popular ones, and to get to other posts you need to click the "Next" button.
That is, when I visit the main page using this code with BeautifulSoup

url = " https://www.b17.ru "
Site = rq.get(url)

I receive only popular articles.
Is it possible to do something like this:

if you can still click the "next" button:
click the "next" button
else:
save the page

Answer 1

Look through the debug console in the browser what request goes to the site after clicking "Next". Repeat it in your program.