N
N
not_eugen2019-08-11 00:20:43
bash
not_eugen, 2019-08-11 00:20:43

How can I find pages that contain certain text from a list of urls?

Hello, I have the following task:
There is a txt file with a list of url types (100k lines) without http:// and https://

site.com/123.php
site2.info/456.php

I need to save to a separate file only those urls in the source code of the pages that contain certain text
For example "Statistic and a Parameter"
I tried to implement this task using a .sh script, because I need to run the script on Ubuntu, but I could not figure it out with curl.
Thank you in advance for your help!

Answer the question

In order to leave comments, you need to log in

3 answer(s)
S
Saboteur, 2019-08-11
@saboteur_kiev

something like this?

#!/bin/bash
while read; do
  echo "processing the following string: ${REPLY}"
  if curl ${REPLY} |grep "Statistic and a Parameter"; do
    echo "${REPLY}" >> urls_with_text.txt
  done
done<file_with_urls.txt

A
Andrew, 2017-12-26
@nur2050

<?php
$array = [
[
  'player_id' => 1,
  'count_goals' => 1
],
[
  'player_id' => 1,
  'count_goals' => 1
],
[
  'player_id' => 1,
  'count_goals' => 1
],
[
  'player_id' => 1,
  'count_goals' => 1
],
];

function filter($array)
{
  $result = [];
  	foreach($array as $key => $value) {
    if (array_key_exists($value['player_id'], $result)) {
      $result[$value['player_id']]['count_goals'] += $value['count_goals'];
    } else {
      $result[$value['player_id']] = $value;	
    }
  	}
  return array_values($result);
}

print_r(filter($array));

N
nur2050, 2017-12-26
@nur2050

Andrey
in the course of work noticed that when there are more values ​​in the array than 2 i.e.:

$array = [
[
  'player_id' => 1,
  'count_goals' => 1
],
[
  'player_id' => 1,
  'count_goals' => 1
],
[
  'player_id' => 1,
  'count_goals' => 1
],
[
  'player_id' => 1,
  'count_goals' => 1
],
];

the result is different:
array(
  (int) 0 => array(
    'player_id' => '1',
    'count_goals' => (int) 3
  ),
  (int) 1 => array(
    'player_id' => '1',
    'count_goals' => (int) 1
  )
)

what could be wrong, thanks

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question