How to remove duplicate ip addresses from a file?

Question

Perl

there is a file with a list of addresses in it, how to remove identical ip addresses using grep or other tools so that they do not exist at all? That is, so that only unique ones remain.

$ cat access.log
212.164.35.62:80
212.164.35.62:80
212.164.35.62:443
212.164.3.95:80
212.164.4.26:80
212.164.4.26:80
212.164.4.26:443
212.164.48.174:80
212.164.49.104:80

Answer 1

Then like this:
cat access.log | awk '{split($0,a,":"); print a[1]}' | sort-u

Answer 2

cat access.log | sort | uniq

Answer 3

awk -F: '{print $1}' access.log |uniq -u

Answer 4

cat ip.txt | cut -d ':' -f 1 | sort -u

Answer 5

cat access.log | awk -FS: '{printf $1 "\n"};' | sort | uniq
This is not a problem for you if you cut out the colon and the port number.
If there is a problem, then the question is, what port with such entries

212.164.35.62:80
212.164.35.62:443

should stay?

Answer 6

thanks Konkase

#!/bin/bash

for ip in `cat ./access.log | awk '{split($0,a,":"); print a[1]}'`
  do
  dup=`cat ./access.log | awk '{split($0,a,":"); print a[1]}' | grep $ip | wc -l`
    if 
      then
        : 
      else
        echo -e "$ip" 
    fi
done

Answer 7

perl -i -ne 'print unless $a{$_}++' access.log