Parsing a Site in Ruby?

Question

ruby

wrote a site parser for a product, but it takes data when you specify a link to a specific product, but it is necessary that he himself run through the entire site and collect data. Who can help?

Answer 1

Why not write another parser that looks, for example, in the sitemap of the site and collects all relevant links (ie links to the product card).
At the moment when the link is needed, we create a task in delayed_job (for example), and the task itself simply executes the script that is already ready (takes a link as input).
I don't think it's worth mentioning the advantages of using queues. :)

Answer 2

either give him the entire list of products, or make him parse it himself from the site.