How to find out the encryption method if there is an encrypted and decrypted file?

Question

Encryption

unencrypted file
encrypted file

Answer 1

No way, because this is a special case of the encryption function.

Answer 2

Well, if the files are identical in size - you can start with xor between them - it is not at all excluded that the output will be some kind of readable phrase like "Pisya Kamushkin is a megahacker!")

Answer 3

only in non-traditional ways

Answer 4

It could be overkill . Since there are only a finite number of known encryption algorithms in the world, there is a decent chance that one of them was used. Of course, it is not 100%, but usually there is no point in reinventing the wheel, which will work even worse, so the chance is good.
The problem is that for enumeration you also need a key (ie, in simple words, a password). Without a key, it will not be possible to check even one of the known algorithms, because you will need to sort through all kinds of keys, and there are too many of them (not for simple algorithms).
It was a head-on approach, i.e. based on the assumption that the encrypted file is the real onecipher, that is, a complete unreadable mess. However, there is a chance, and judging by the screen, a decent one, that the cipher contains some hints, patterns, hints about what properties the cipher has. You can study them and build on them.
For example, you can see the frequency of occurrence of different characters. Another question is what to consider as a character in this analysis, but usually it is 1 byte. It may turn out, for example, as it often happens, that there are noticeably more zeros. Perhaps this is not completely hidden information in the source file, or it is a property of this particular encryption algorithm. There is room for conjecture.
Or you can see if and how readable characters are grouped, if they make up words, or if they are similar in length to something. How to interpret this is already a task for your imagination. Just if it's not random, then it's a hint that you can use.
Well, then you can continue to meditate, think, look, guess, until an insight (enlightenment) happens. And a series of such insights can lead you to the answer, what kind of algorithm is this. But there is no guarantee. All patterns found may not be sufficient to find an answer, and in the worst case, the author of the cipher simply feeds you these patterns so that you have something to occupy your mind. After all, the author of the cipher is unlikely to want you to learn how to extract unencrypted data so easily.
In conclusion, I note that the most reliable will be to analyze (decompile) an application that encrypts, if you have access to it. But this is a topic beyond the scope of your question and this answer.

Answer 5

Usually, they analyze not the files that were encrypted, but the program that did it. As it was rightly noted here, the overwhelming majority use ready-made algorithms and their implementations. And this allows us to search for certain sequences in the program. Namely, algorithms have a specific initial state, it must be set at the time of initialization, and therefore we can find it in the program. Also, in order to optimize speed, the main encryption blocks are written almost in assembler and almost do not differ from compiler to compiler, they can also be identified.
There was just such a plugin in the PEiD tool.