Are there any applications that generate point cloud animation from video?

Question

3D

in order to submit a video to the input on which, for example, objects are moving in the foreground - and at the output to get a 3d animation of a point cloud?

It is clear that neural networks are needed here.

Recently, somewhere I saw information about an application that makes a 3D figure of a face from a photo of a face
, maybe there is such a thing for a video?

Answer 1

The task can be solved in hardware, 3d cameras from the same intel in the examples in the sdk offer software for just this, they give a cloud of points, but it’s impossible to use it
--------
the neural network, those that are available, will not help with the video, the main the reason is that neighboring frames are analyzed independently and all this works with huge errors.
The angle vector of the gaze direction/position of the face will jump like crazy from frame to frame, I haven’t said anything about emotions yet, each emotion is also parameterized, all this will not allow to give an exact match of the same point on the face in adjacent frames, i.e. having built, for example, surfaces based on these points, they simply will not converge on the next frame, as a result there will be strong noise that makes it difficult to use at least some adequate application (I played with 3d cameras from intel, the strong noise of the result makes everything meaningless)

Answer 2

Theoretically, all this can be achieved without neurons, purely through computer vision and a pinch of mathematics. Blender3d has support for motion capture through marking points on the video and subsequent tracking through relative speed ( example ). It is possible to write a script through the built-in python interpreter to approximate tracked points. Accuracy is likely to be poor, and it may be necessary to do multiple runs with different combinations of markers and a subsequent combination of results.
And a face from a static photo can only be roughly guessed, again, most implementations now rely on computer vision.