B
B
brunot2015-12-22 16:07:33
PHP
brunot, 2015-12-22 16:07:33

How to shuffle lines in a large file (up to 1.5GB) using php?

In general, I dug up the script on the forum, it interferes with the rules, but with a large file size, some kind of limit is triggered somewhere, while:
5118f538b40b444eb4b66ca56f1cd534.png(Denver on Windows)
The question is rather like this: where else do you need to turn off the limit to make it work?
The script for the mix itself is as follows:

<?php 

ini_set('memory_limit', '-1');
ini_set('max_execution_time', '-1');

$start = microtime(true);
define('NL',chr(13).chr(10));  
$file='file.txt'; 
$arr=explode(NL,file_get_contents($file));  
$f=fopen($file,'w'); 
shuffle($arr);  
fputs($f,implode(NL,$arr));  
fclose($f);  

$time = microtime(true) - $start;
printf('Скрипт выполнялся %.4F сек.', $time);

?>

What he writes:
4b5f57fb8c3141d19b61c08bd616db1c.png
Also do not mind considering other suitable solutions

Answer the question

In order to leave comments, you need to log in

4 answer(s)
W
wol_fi, 2015-12-22
@brunot

You have some wrong bees $arr=explode(NL,file_get_contents($file));- it's bad for any language, not just php, to read the entire file into memory.
Here is an example for shuffling a file, without fully downloading it, using line positions.

S
Saboteur, 2015-12-22
@saboteur_kiev

Why not use SQL? Drive rows into SQL, and randomize only indexes, or a separate column with a row number. It will work much faster. Random access to the desired string is also faster. Memory will not consume at all. Scaled.

R
romy4, 2015-12-22
@romy4

well or without SQL.
Stuff an array of strings (from 0 to the number of lines in the file). shuffle this array, then read one value from the array, read this line number, write to a new file.
Don't forget to save the positions of the beginning of the lines, relative to the beginning of the file. 10 minutes of work.

N
Nicholas, 2015-12-22
@healqq

Do not interfere with the entire file, divide it into an adequate number of parts.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question