A
A
Alexey Yarkov2017-01-14 22:40:09
linux
Alexey Yarkov, 2017-01-14 22:40:09

How to deal with the date?

There is a site that once every 3-5 days gives a list of proxies to this URL:
http://site.com/proxy/list_{dd.mm.YYYY}.txt
I need to do this:
Send a request to the url with today's date
If the status is 404, then reduce the date by 1 and send the request again and so on until I get the status 200
Bye there is this code:

get_proxy_list() {
  savefile=~/.PROXY_LIST
  date=$(date +%d.%m.%Y)
  url="http://site.com/proxy/list_${date}.txt"
  # Вот тут надо вкорячить while status -ne 200
  status=$(wget --spider -S ${url} 2>&1 | grep "HTTP/" | awk '{print $2}')
  if [ "${status}" -eq 200 ]; then
    wget --output-document=${savefile} ${url} 2>&1
    cat ${savefile}
  elif [ -f ${savefile} ]; then
    cat ${savefile}
  else
    echo "Proxy list not found"
  fi
}

Answer the question

In order to leave comments, you need to log in

2 answer(s)
A
Alexey Yarkov, 2017-01-15
@yarkov

It all ended up like this:

__get_date() {
  local date=$(date --date="-$1 day" +%d.%m.%Y)
  echo ${date}
}

__get_url() {
  local date="$1"
  echo "http://webanetlabs.net/freeproxylist/proxylist_at_${date}.txt"
}

__get_status() {
  local offset="$1"
  local date=$(__get_date ${offset})
  local __url=$(__get_url ${date})
  local status=$(wget --spider -S ${__url} 2>&1 | grep "HTTP/" | awk '{print $2}')
  echo "${status}"
}

get_proxy_list() {
  local savefile=~/.PROXY_LIST
  local counter=0
  while [ $(__get_status ${counter}) != "200" ]; do
    counter=$(($counter + 1))
  done
  local date=$(__get_date ${counter})
  local url=$(__get_url ${date})
  wget --output-document=${savefile} ${url} > /dev/null 2>&1
  cat ${savefile}
}

A
abcd0x00, 2017-01-15
@abcd0x00

Remake it first

get_proxy_list() {
  savefile=~/.PROXY_LIST
  date=$(date +%d.%m.%Y)
  url="http://site.com/proxy/list_${date}.txt"

  ...

}

In it
make_dmy_date() {
    date "+%d.%m.%Y"
}

make_proxy_url() {
    echo "http://site.com/proxy/list_$(make_dmy_date).txt"
}

make_proxy_ofname() {
    echo "~/.PROXY_LIST"
}

get_proxy_list() {
  url=$1
  savefile=$2

  ...

}

get_proxy_list `make_proxy_url` `make_proxy_ofname`

And then write the functions you need. If you need to decrement a date, write a function to do so. At the same time, there is control over the code inside the function, but the code itself can turn out to be non-trivial, since when decrementing the date, you need to take into account that it can reach zero and this will also need to be covered with some kind of code.
And if you write everything in a heap, then there will be such a dump as you have now - clumsy and unsuitable for some subtle changes and settings.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question