Optical 3d scanning, point cloud has perspective distortions, how to compensate for them?

Question

Mathematics

Hello.
I accidentally got a 3D scanner with a turntable that scans with a video camera and special lighting, in the form of a changing chessboard. The stock software works well, but it's not perfect.
I'm interested to figure it out myself and maybe get a better quality result.
After each scanning session, a set of 16 png photos and 16 sets of point coordinates is obtained.
Photos 640x480 pixels, each set of coordinates contains 640x480 = 307200 three-dimensional coordinates.
I took the c++ opengl example, added reading png and reading coordinates from a file.
As a result, one scan frame in opengl looks like this:

If you move the camera a little, then like this:

Side view:

The problem is that this is a cylinder, and opengl is set to orthogonal projection, that is, it does not introduce distortion.
This is the cylinder itself scanned with perspective distortion.
If I understand correctly, to convert an orthogonal projection into a perspective one, the corresponding matrix is ​​used. And apparently it is possible to reverse the transformation, the inverse matrix?