I
I
ImNotANoobby2021-12-07 16:13:37
Python
ImNotANoobby, 2021-12-07 16:13:37

How can I parse a page with all products, if after clicking the "show all products" button, the link does not change?

When I read all product links on a page, only those that are displayed are parsed for me. On the page, it is possible to choose to display all products at once, but this action does not change the link. How can I get a page with all the products?

<code lang="python">
import requests
from bs4 import BeautifulSoup

url = 'https://somesite.com'
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
r = requests.get(url, headers=headers)
r.encoding = 'utf-8'
print(r.encoding)
 
 
soup = BeautifulSoup(r.content, 'html.parser')
res = soup.select(".block_product")


for a in res:
   name = a.select('a')
   price = a.select('.jshop_price span')

   # print(a[0].text)
   print(name[0].text)
   print(price[0].text)
</code>

Answer the question

In order to leave comments, you need to log in

1 answer(s)
I
ImNotANoobby, 2021-12-07
@ImNotANoobby

I went to the inspector, selected this button to change the number of displayed products.

<select id="limit" name="limit" class="inputbox" onchange="submitListProductFilters()">
  <option value="5">5</option>
  <option value="10">10</option>
  <option value="12">12</option>
  <option value="15">15</option>
  <option value="20">20</option>
  <option value="25">25</option>
  <option value="50">50</option>
  <option value="99999" selected="selected">Все</option>
</select>

and added to url ?limit=99999
Earned

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question