How to check the list of links for availability?

Question

Monitoring

I have a list of urls (and urls are internal pages of sites, not main pages), about 5000 pieces, and a modest server on centos. The task was to check these 5k URLs several times a day (ideally, once an hour) for availability, and if suddenly one of the URLs gives a 404 error, I should immediately receive an alert, for example, by mail.
Recommend a solution. At the beginning I was looking for some script with a web-face, where everything would be convenient and beautiful, but I found only scripts for monitoring resources in general, which check only the main page of the site, which does not suit me at all. Maybe I was looking for a bad panel?
Also, I think that it is quite possible to do this on a bash script that would go through all the urls in the links.txt file and if any of the urls gives a 404 error, send me an email. I won’t write this on my own, if there are suitable examples, I will be grateful if you show me.
The main thing is that the solution is simple and not difficult.
Thanks in advance)

Answer 1

The script will be simple...
Beginning of the loop
Read a line from the file
Check the read link using curl
If 404 - send soap to the admin
if the end of the file is stop, otherwise to the beginning of the loop...
Encoding all this is lazy

Answer 2

it seems to me that you can try to parse the responses of the curl utility, it returns the web server response code there

[email protected]:~$ curl -I https://toster.ru
HTTP/1.1 200 OK
Server: QRATOR
Date: Mon, 24 Jul 2017 14:56:07 GMT
Content-Type: text/html;charset=utf-8
Connection: keep-alive
Keep-Alive: timeout=15
X-Powered-By: PHP/5.6.20-1+deb.sury.org~trusty+1
Content-Encoding: gzip
Strict-Transport-Security: max-age=31536000
Public-Key-Pins: pin-sha256="klO23nT2ehFDXCfx3eHTDRESMz3asj1muO+4aIdjiuY=";pin-sha256="ATPF8U6AdEXM7aD9/PTAJldZj9jI6NWEvRGMbDJiN3g="; max-age=15552000

interested in the string HTTP / 1.1 200 OK
well, the cron script is polled once an hour