Is the architecture of the system for parsing done correctly?

Question

Python

The python script walks the site and collects links, then it sends them to Rabbitmq. Other python scripts access the queue and parse all the necessary data and put it in the mongo.
Actually the question is, is it possible to do something better in this system, or can you use some other tools?

Answer 1

Normal solution. There are no ideal solutions.

Answer 2

You need to be aware that now many sites come to life only if JavaScript is enabled.
That is, not everything can be parsed on bare Python.
To fully read information from many sites, you need something like SimplerJS, PhantomJS, Selenium