How to scrape many different sites?

Question

Python

There is a spider that needs to go from one general page to pages with different domains. The role of start_urls will always be one page. How to specify allowed_domains? registering domains there is not an option, there can be a lot of them.

Answer 1

the problem was that I specified allowed_domains = ["domain"], and therefore the spider did not go to external resources. specifying allowed_domains = [] in this way solved the problem.