Answer the question
In order to leave comments, you need to log in
How to shuffle lines in a large file (up to 1.5GB) using php?
In general, I dug up the script on the forum, it interferes with the rules, but with a large file size, some kind of limit is triggered somewhere, while:
(Denver on Windows)
The question is rather like this: where else do you need to turn off the limit to make it work?
The script for the mix itself is as follows:
<?php
ini_set('memory_limit', '-1');
ini_set('max_execution_time', '-1');
$start = microtime(true);
define('NL',chr(13).chr(10));
$file='file.txt';
$arr=explode(NL,file_get_contents($file));
$f=fopen($file,'w');
shuffle($arr);
fputs($f,implode(NL,$arr));
fclose($f);
$time = microtime(true) - $start;
printf('Скрипт выполнялся %.4F сек.', $time);
?>
Answer the question
In order to leave comments, you need to log in
You have some wrong bees $arr=explode(NL,file_get_contents($file));
- it's bad for any language, not just php, to read the entire file into memory.
Here is an example for shuffling a file, without fully downloading it, using line positions.
Why not use SQL? Drive rows into SQL, and randomize only indexes, or a separate column with a row number. It will work much faster. Random access to the desired string is also faster. Memory will not consume at all. Scaled.
well or without SQL.
Stuff an array of strings (from 0 to the number of lines in the file). shuffle this array, then read one value from the array, read this line number, write to a new file.
Don't forget to save the positions of the beginning of the lines, relative to the beginning of the file. 10 minutes of work.
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question