B
B
Bur Ov2018-09-18 11:41:01
linux
Bur Ov, 2018-09-18 11:41:01

How to quickly process 10gb txt files with php methods?

Good afternoon.
There was such a need to process 10GB of text files, all these files are in different folders. At the moment, I wrote a php script that runs through all the folders and finds only .txt files there, then it opens each file and reads it line by line and looks for keywords in the lines (a piece of code indicated)

foreach ( $array_url as $addr ) {
    $handle = @fopen($addr, "r");
    if ($handle)
    {
      while (!feof($handle))
      {
        $buffer = fgets($handle);
        foreach ( $searchthis as $search ) {
          if(strpos($buffer, $search) !== FALSE)
            $matches[] = $buffer;
//Тут мне нужно хотя бы 1 вхождение, но скрипт ищет все вхождения.
//Как при нахождении хотя бы 1 вхождения сразу перейти в другой файл?
        }
      }
      fclose($handle);
    }
    unset($buffer);
    unset($handle);	
  }

Then I look, if there is at least something in $matches, then I transfer this file to another folder, and so on for all txt.
This all works well on ~100 files (file size up to 1mb), but with 10GB the file will not work like that...
What is the best way to process such a volume? Maybe it will be better to process only by means of linux?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
S
Stalker_RED, 2018-09-18
@burov0798

https://ru.wikipedia.org/wiki/Grep
-r- search recursively in subfolders
--include=\*.txt- only .txt files - search
'hello world'phrase (not a phrase, but a regular expression)
/path/to/dir- where to start the search
-l- Displays only the names of files containing matched lines, one per line. If the pattern is found on multiple lines of a file, the file name is not repeated.
I'm probably not going to reprint the entire certificate here.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question