How to parse an AJAX page in Ruby?

Question

ruby

I use the curb library for page download and nokogiri for parsing. The content is updated by the “load more” button, so I got the idea to parse the current url, click on the button, parse the new url, and so on until the button disappears. The problem is that the libraries used do not know how to click, so please tell me the tool, or maybe a different approach to the solution than mine. Thank you.

Answer 1

To click buttons you need a browser, you can try Capybara + PhantomJS. Or see what requests AJAX makes on the button and repeat until an empty response.