What technologies are there for video recognition (namely video, not individual frames)?

Question

Pattern recognition

There is OpenCV, it can recognize individual video frames. I can recognize the person in every frame. But what if I need to know if he is running or waving his arms? I can recognize the tree in every frame, but I need to figure out if the tree is swaying in the wind. I can recognize the wind generator in every frame, but I need to know if it is spinning right now.

Therefore, the question arises: do technologies, libraries, algorithms, or at least concepts for object recognition exist for a certain period of time? For example, I have a series of several frames, each of which definitely has a person, and I need to find out if he is walking or waving his arms.

Maybe there are at least some articles or descriptions of the ideas of such technology? Ideas for training such a system, data storage and so on.