How to work with large arrays?

Question

PHP

What is the right way, for example, to find common values ​​in huge arrays (100-500k values) so that it does not take up a lot of resources?
If you do this in the usual way, then the output is a very loaded process. Maybe you should choose another language (I don't know how php is adapted for such tasks) or design it differently, for example, divide it into many small arrays?

$arr1 = [];
$arr2 = [];

for($i = 0; $i <= 400000; ++$i) {
  $arr1[] = rand(1000000, 100000000);
  $arr2[] = rand(1000000, 100000000);
}

$out = array_intersect($arr1, $arr2);

Answer 1

Well, another structure suits well for such cases - sets. They are in python, for example. They work much faster due to the disorder and uniqueness of values.

Answer 2

php has SplFixedArray - it will eat less memory. IMHO you are trying to hang on php what it is not supposed to do. I would implement this logic on the side of the one who stores this data, for example mysql. If you need to find intersections of sets, it is very convenient to use intersections in redis .

Answer 3

Add values ​​like $arr[$i] = true and do key-based intersection, array-intersect-key seems to be. In fact, this is the implementation of the set.

Answer 4

write in any language (well, or a library - for Theano python) that supports threaded processor operations, taking into account the work of these very operations.

Answer 5

I tried to do the same in python on the advice of lyeskin , maybe wrong. The result is better, of course, but still I would not call it satisfactory.

import random
set1 = set()
set2 = set()
i = 0

while i < 400000:
        set1.add(random.randint(1000000, 10000000))
        set2.add(random.randint(1000000, 10000000))
        i = i+1;

print set(set1).intersection(set2)

Answer 6

Sort and write a linear algorithm to intersect sorted arrays . array_intersect most likely works as a square and therefore loads everything.