E
E
Emil Revencu2016-03-06 21:40:24
Python
Emil Revencu, 2016-03-06 21:40:24

How to load website via Mechanize or GRAB (Python 2.7.11)?

How to load site https://www.iec.co.il/pages/billspayment.aspx via Mechanize or GRAB (Python 2.7.11)?
My GRAB code:

from grab import Grab
header={
'Host':'www.iec.co.il',
'Referer':'https://www.iec.co.il/pages/billspayment.aspx',
'Upgrade-Insecure-Requests':'1',
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36'
}
g = Grab()
url='https://www.iec.co.il/pages/billspayment.aspx'
g.setup(headers=header)
g.go(url)
g.doc.save('result.html')

Mechanize:
import mechanize
br = mechanize.Browser() 
br.set_handle_robots(False)
br.set_handle_refresh(False) 
br.addheaders = [
('Host','www.iec.co.il'),
('Referer','https://www.iec.co.il/pages/billspayment.aspx'),
('Upgrade-Insecure-Requests','1'),
('User-Agent','Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36')
] 
url='https://www.iec.co.il/pages/billspayment.aspx'
response = br.open(url)
s=response.read()
f = open('result.html', 'w')
f.writelines(s)
f.close()

But neither option returns the full page
. How can I get the full page (Mechanize, GRAB)?
Through Selenium I get, but it takes a long time to load

Answer the question

In order to leave comments, you need to log in

2 answer(s)
D
Dimonchik, 2016-03-06
@dimonchik2013

compare the readings of fiddler and for the browser and Python
, try to emulate the request performed by JS scripts, if there is one, it
most likely returns everything, you just need to pull the accompanying JS scripts, such as the one below

<script src="/c3650cdf-216a-4ba2-80b0-9d6c54670b0b105e58d2-ea0f-484e-b88c-0e2c1499ec9bd71e4b42-4498-44e3-89b6-fa43b6845326" type="text/javascript"></script>

E
Emil Revencu, 2016-03-06
@Revencu

If I add a Cookie string taken from the Request Headers browser to HEADERS, it returns the entire page, but the next day it crashes again.
Same problem :
www.cyberforum.ru/java-networks/thread1547066.html
forum.kodi.tv/showthread.php?tid=247154

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question