How to check hash sum of files in linux?

Question

linux

Good afternoon, there are about 500 txt files in the directory, some of the contents are similar, you need to check the hash of the sum to identify similar files, how to implement this?

Answer 1

The hash sum will be the same only for absolutely identical files. Similar to her not to be tracked

Answer 2

You can compare text files by content using the diff utility.
https://www.opennet.ru/man.shtml?topic=diff&catego...

Answer 3

You can use fuzzy hashes like ssdeep

Answer 4

Be specific with the meaning of "similar".
Similar in meaning and similar minus, say, - the number of spaces - these are different things ....
You can only determine exact matches with hash sums.

Answer 5

The sum hash can be calculated using md5sum (fast but collisions are possible) and sha256sum (considered reliable). There are other implemented algorithms but they are not commonly used.
The same hash sum is only with a complete match of the files: one bit changed and the hash sum is completely different. Collisions are possible with md5 (two files correspond to the same hash), but the chance of running into this is vanishingly small.
On the top, this can be done like this:

find -name  "*txt" -exec sha256sum {} \; | sort | uniq -D -w 65