Perl how to speed up a script?

Question

Perl

We need to compare each element of the first array with each element of the second one and print unique and non-unique values ​​to different files
#!/usr/bin/perl
.....
foreach $nn (@array0) {
$g=0;
foreach $mm (@array1) {
if ($nn==$mm) {
$g=$g+1
print FILE1 "$nn\n";
}
}
if ($g==0) {
print FILE2 "$nn\n";
}
}
......
How to speed up the script? processing 10,000 x 1,000,000 = 10 min
and you need to compare approximately 1,000,000,000 x 1,000,000,000,000 ...... tell me the approach
If you use for, then 10,000 x 1,000,000 = 37 min
I specify
Comparison does not slow down much. ....
I ran the following program without comparison:
foreach $nn (@array0) {
foreach $mm (@array1) {
$g=$g+1
}
}
as a result, the running time is almost 10 minutes
, how can I compare the values ​​differently, not through foreach or for?

Answer 1

You can use the ready-made module:
search.cpan.org/~zmij/Array-Utils-0.5/Utils.pm
The module is fast.
More or less like this.
There are two arrays. We compare them and display unique values:

my @a = qw( a b c d );
my @b = qw( c d e f );
my @c = array_diff( @a, @b );
say for (@c)

Result:

a
b
e
f

Now we find elements that are not unique.

use feature 'say';
use Array::Utils qw(:all);


my @a = qw( a b c d );
my @b = qw( c d e f );
my @c = intersect( @a, @b );
say for (@c)

Result

c
d

Answer 2

1. Simple and fast algorithms are in Perl Cookbook
2. Arrays of a billion and a trillion elements cannot be compared like that - there is simply not enough memory.
3. What type of data? It is clear that in Perl this does not seem to be so important, but it matters for solving the problem.
Offhand solution:
Let's say that we only have integers - values ​​​​from 0 to 65535. Let's build a bit mask of the numbers available in the array, and if there is a number, we will set the corresponding bit to 1. The size of the mask is obviously 65536 bits or 8192 bytes, which is not at all lot.
So, go through the first array and fill in the mask.
Now we go through the second array and if the bit in the mask for the current number is set to 1, then the number is not unique.