How to test protocols for vulnerabilities?

Question

Computer networks

There is a task - to test the data transfer protocol. How do they test protocols in general, how do they look for vulnerabilities?
There is evidence that AES-256, RSA are used for encryption

Answer 1

Read about fuzzing, for example: https://en.wikipedia.org/wiki/Fuzzing

Answer 2

Typically, protocols are verified and then modeled.