E
E
ekaterina_cat2020-01-12 00:00:20
Python
ekaterina_cat, 2020-01-12 00:00:20

How to get a list of valid files from a list of url files?

Hello. There was a task from the list of the form (about 5000 in total):


mysite1.ru/upload.zip
mysite2.ru/dle.gzip

Get a list of archives available for download - by 200 response or not, it doesn't matter.
What would the output be a txt file with links available for download.
A multithreaded solution is desirable. To periodically run from the north.
But the problem is that some of my links are with http and https and www and without www and hence the redirect.
There are false answers that the file is not available - although it is available.
What I found on curl. But it doesn't write to a file and I can't multithread it.

#!/bin/bash
while read LINE; do
curl -o /dev/null --silent --head --write-out "%{http_code} $LINE\n" "$LINE"
done < url-list.txt

Thanks for the replies!

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question