How to speed up the link checker?

Question

Python

Good day. I have a script that takes a link from a file and checks it for certain content in an html page. The problem is that the script works quite slowly and is not efficient at all with large databases (several million links). Can anyone improve the code or throw off your script?

https://pastebin.com/sP97c4uL

Answer 1

At the very least, I advise you to parallelize the threads if the processor cores and network bandwidth allow. I'm not sure that I can find the code, but I wrote a script that allows you to download pictures from one portal at a certain range of url addresses. Parallelization really helped.
UPD.
I found the code, I can not vouch for its performance with respect to the current service, but "honestly, everything worked."
https://pastebin.com/grFejrjR
There is a lot of extra code that is not related to the task. At a minimum, there is a proxy change and their check, the lists of which I downloaded from free sites and checked online. And extra "beauty" in the form of a colorful output. Yes, these are the very pictures that someone uploads through the "Lightshot" program :)

Answer 2

asyncio or golang