V
V
Vlad_beg2019-07-20 20:57:01
Python
Vlad_beg, 2019-07-20 20:57:01

How to scrape a dynamic website with Scrapy?

Please tell me why, when parsing some links, the content is parsed normally, while others return `None` for all fields, looking at the `Network` tab in the browser, I see that the content is returned in `JSON` format. How to parse content from this site?
Thanks in advance!

class DynamicSpider(scrapy.Spider):
name = 'products'
url = [
    'https://www.lazada.sg/products/esogoal-tactical-sling-bag-outdoor-chest-pack-shoulder-backpack-military-sport-bag-for-trekking-camping-hiking-rover-sling-daypack-for-men-women-i204814494-s353896924.html?mp=1',
    'https://www.lazada.sg/products/esogoal-selfie-stick-tripod-extendable-selfie-stick-monopod-with-integrated-tripod-and-bluetooth-remote-shutter-wireless-selfie-stick-tripod-for-cellphonecameras-i205279097-s309050125.html?mp=1',
    'https://www.lazada.sg/products/esogoal-mini-umbrella-travel-umbrella-sun-rain-umbrella8-ribs-98cm-big-surface-lightweight-compact-parasol-uv-protection-for-men-women-i204815487-s308312226.html?mp=1',
    'https://www.lazada.sg/products/esogoal-2-in-1-selfie-stick-tripod-bluetooth-selfie-stand-with-remote-shutter-foldable-tripod-monopod-i279432816-s436738661.html?mp=1',
]

script = """
    function main(splash, args)
      assert(splash:go(args.url))
      assert(splash:wait(2.5))
      return {
        html = splash:html()
      }
    end
"""

def start_requests(self):
    for link in self.url:
        yield SplashRequest(
            url=link,
            callback=self.parse,
            endpoint='execute',
            args={'wait': 1.5, 'lua_source': self.script},
            dont_filter=True,
        )

def parse(self, response):
    yield {
        'title': response.xpath("//span[@class='pdp-mod-product-badge-title']/text()").extract_first(),
        'price': response.xpath("//span[contains(@class, 'pdp-price')]/text()").extract_first(),
        'description': response.xpath("//div[@id='module_product_detail']/h2/text()").extract_first()
    }

Answer the question

In order to leave comments, you need to log in

1 answer(s)
D
Dimonchik, 2019-07-20
@dimonchik2013

https://habr.com/en/post/312816/

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question