Answer the question
In order to leave comments, you need to log in
How can I parse a page with all products, if after clicking the "show all products" button, the link does not change?
When I read all product links on a page, only those that are displayed are parsed for me. On the page, it is possible to choose to display all products at once, but this action does not change the link. How can I get a page with all the products?
<code lang="python">
import requests
from bs4 import BeautifulSoup
url = 'https://somesite.com'
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
r = requests.get(url, headers=headers)
r.encoding = 'utf-8'
print(r.encoding)
soup = BeautifulSoup(r.content, 'html.parser')
res = soup.select(".block_product")
for a in res:
name = a.select('a')
price = a.select('.jshop_price span')
# print(a[0].text)
print(name[0].text)
print(price[0].text)
</code>
Answer the question
In order to leave comments, you need to log in
I went to the inspector, selected this button to change the number of displayed products.
<select id="limit" name="limit" class="inputbox" onchange="submitListProductFilters()">
<option value="5">5</option>
<option value="10">10</option>
<option value="12">12</option>
<option value="15">15</option>
<option value="20">20</option>
<option value="25">25</option>
<option value="50">50</option>
<option value="99999" selected="selected">Все</option>
</select>
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question