C
C
cegthgtlhj2019-10-16 14:42:55
Python
cegthgtlhj, 2019-10-16 14:42:55

Website parsing in python. How to specify POST request parameters?

There is a website with a search form https://www.sf-filter.com/en/vergleichsliste/?sf-c... => see picture.
I enter a keyword into the form and try to parse the returned query results.
There are a couple of thousand keywords, so you need to do it programmatically.
After pressing the "Submit / Search / Enter" button, the browser shows that there were two POST requests.
The first POST request is like this
https://www.sf-filter.com/WebPortal/async.aspx
with parameters

txt_ArtSearch=0110d0
crypt=024YX9/HsQgaKQQ55DhXTgEqA==WbpRaFfFOkPuQGn0B1jd6g8P0zQUbI06LhmJMoT3xUAHeqUwhFBvqIwAnmxp4lPC
originalUrl=https://www.sf-filter.com/en/vergleichsliste/?sf-client=20

and the second POST request is
https://www.sf-filter.com/id/5F59032AF1A249C9AEA21A4DCED078CF8E2611E9708A4486AE51411487E24F64.htm

with parameters
txt_ArtSearch=0110d0
originalUrl=https://www.sf-filter.com/en/vergleichsliste/?sf-client=20

search form code below
<form action="javascript:void(0)" id="SPCompareList" method="post" name="SPCompareList" novalidate="novalidate">
      <div class="section-inner text-center">
       <h2>
        Cross reference list
       </h2>
       <p class="section-intro">
        Please enter any filter number and we show you the best suitable SF quality filter for your needs.
       </p>
       <div class="form-group" id="searchBtn">
        <input autofocus="" class="flat" id="txt_ArtSearch" name="txt_ArtSearch" placeholder="Enter item number" type="text" value="0110d0"/>
        <span class="form-icon" onclick="$('#SPCompareList').submit()">
         <i class="fa fa-search">
         </i>
        </span>
       </div>
      </div>
     </form>

q04.png
with further manual input of keywords and clicking on the search button, only one post request is added, similar to the first one, but the parameter
crypt=024YX9/HsQgaKQQ55DhXTgEqA==WbpRaFfFOkPuQGn0B1jd6g8P0zQUbI06LhmJMoT3xUAHeqUwhFBvqIwAnmxp4lPC

changes every time
Which direction to look for ?
What is the name of this defense?
Is it possible to somehow bypass it and automate the process without SELENIUM?

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question