What hash is better to get from a file?

Question

Java

We are writing a program to search for duplicates, which hash to use for further checking files, we need speed of work, using MD5 a file of 20 GB was checked for 10 minutes

Answer 1

For large files, consider the hash of not the entire file, but, for example, the initial megabyte or a tricky sequence - ten pieces of megabytes from different parts of the file. Anyway, if you suspect a duplicate, then you need to double-check with another hash or direct comparison.