How to correctly upload files from http?

K

KALT882018-04-23 04:55:12

bash

KALT88, 2018-04-23 04:55:12

Actually there is a host with heavy files and a bunch of directories, no ftp, only password-protected http, to which access will also end soon.

We urgently need to move this whole thing somewhere in the cloud, only about 5TB.
There is a vps with sssh and rdp, unlimited google drive training and dropbox trial.
Specifically, .tar.gz archives are needed, which are located in such directories:
./zone_file/2017-11-21/biz/

http://domain.com/zone_file/2016-01-28/biz/biz.tar.gz.0000
http://domain.com/zone_file/2017-11-21/biz/biz.tar.gz.0000
http://domain.com/zone_file/2017-06-29/net/net.tar.gz.0000
http://domain.com/zone_file/2017-06-29/net/net.tar.gz.0001
http://domain.com/zone_file/2017-06-29/net/net.tar.gz.0002

From 1 to 18 pieces of archives in a directory.
There is a complete list of date folders like /2017-06-29/
And the beginning of the archive name and part of the directory like /biz/
In fact, there are 2 variables and they are known.
There are also html's with the contents of the directories, from which you can get the names of archive files.
It seems that you need to generate paths to html, parse the names of the archives from the existing ones.
Save the links to a list/lists, which will then be transferred to the cloud using wget/curl/bash, creating a directory and a filename from the url, so that later you don’t parse the full mess in the cloud.
For example 20170629net.tar.gz.0000

<html>
<head><title>Index of /zone_file/2017-06-29/net/</title></head>
<body bgcolor="white">
<h1>Index of /zone_file/2017-06-29/net/</h1><hr><pre><a href="../">../</a>
<a href="net.tar.gz.0000">net.tar.gz.0000</a>                                    29-Jun-2017 19:50    200M
<a href="net.tar.gz.0001">net.tar.gz.0001</a>                                    29-Jun-2017 19:55    164M
</pre><hr></body>
</html>

/zone_file/status/supported_tlds

xn--tckwe,xn--tiq49xqyj,xn--unup4y,zappos,zara,zero,zippo,zip,zone,zuerich,..

Index of /zone_file/
2016-01-13, 2016-01-14, ..., 2018-04-22

Reply

Answer the question

In order to leave comments, you need to log in

1 answer(s)

C

CityCat4, 2018-04-23
@CityCat4

man wget