Is it possible to get JSON from the server of a site that is protected from parsing?

M

Maksimg0lubev2021-05-24 16:00:13

Parsing

Maksimg0lubev, 2021-05-24 16:00:13

Greetings!

At the moment I'm making a web scraper and I'm using Selenium in Python as well as JS , but since the speed of Selenium is not so high within the current project, I have to investigate requests and responses in Networking -> XHR.

I noticed one get request, during which the site receives json with all the necessary data.
But when I do this already through requests.get (), I do not receive the same data (JSON) that the site receives.
As far as I understand, this is protection against parsing.
If this is indeed parsing protection, then how can you get JSON from the server?

Instead of the expected JSON, I get the following response with the md5 hashing JS code:

<!DOCTYPE html>
<html>
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  <meta http-equiv="refresh" content="10;URL=/ciez2a">
</head>
<body>
  <script type="text/javascript">
function md5cycle(e,t){var i=e[0],a=e[1],r=e[2],n=e
var ipp = {
  decrypt:function()
  {
                var decrypt = new JSEncrypt();
                decrypt.setPrivateKey("MIIkKQIBAAKCCAEArbPJ2oT3hgoDGT");
                return decrypt.decrypt("TbNVvQdiXzncfAzNISydUx0w4VUzgfDj05Sv5Xv4mZXcOWIWq");
  },
  setCookie:function()
  {
    document.cookie="ipp_key=" + this.decrypt() + "; path=/;";
    document.cookie="ipp_uid=1621858146575/8SLIJxfpZaJLWSPi/0WagCYVKhhQ/JQnzVMhZow==; expires=Tue, 31 Dec 2030 23:59:59 GMT; path=/;";
    document.cookie="ipp_uid1=1621858146575; expires=Tue, 31 Dec 2030 23:59:59 GMT; path=/;";
    document.cookie="ipp_uid2=8SLIJxfpZaJLWSPi/0WagCYVKhhQ/JQnzVMhZow==; expires=Tue, 31 Dec 2030 23:59:59 GMT; path=/;";
  },
  makeUrl:function(url)
  {
    url += "fa821dba_ipp_key=" + this.decrypt() + "&" + "fa821dba_ipp";
    return url;
        }
};


    (new Fingerprint2).get(function(e,t){
      salt="1766454767";
      document.cookie="ipp_sign="+e+"_"+salt+"_"+md5(e+salt)+"; expires=Tue, 31 Dec 2030 23:59:59 GMT; path=/;";
      ipp.setCookie();
      window.location.href = "https://brandshop.ru/getproductsize/298106/?" + window.location.hash;
    })
  </script>
</body>
</html>

Reply

Answer the question

In order to leave comments, you need to log in

1 answer(s)

R

Rsa97, 2021-05-24
@Rsa97

Until the required cookie (ipp_*) is present in the request, this script will be returned. It sets the cookie and does a page reload.
So see what cookies are needed in the site request and substitute them in your request.