Test stand in the cloud with the installation of proprietary software with a GUI, where to dig?

Question

Software testing

Given

• There is a closed source GUI application with an open API with somewhat "inaccurate" documentation. (hereinafter referred to as software )
  
• There is an open-source python-library that non-trivially (in terms of installation) integrates into this application, and then communicates with it on the web)
  
• The software has very limited control from the command line (or terminal), and basically the only thing you can do is run the project there
  
• We are now thinking about how we can add tests to the library, and I see the most adequate option as building some kind of container in which there will be already running software, with all open windows, a project, etc. Well, then at the next test, you will just have to cut it and test it using the black box method.
  
• It would be nice to test all this at the same time and multi-platform (win / mac / lin)
  

The problem is that I really have no idea where to dig. I'm trying to start it in the docker for now, but it's slow, besides, in parallel there are problems not only with the GUI, but also with the audio (because the software does not work without it). Of course, it would probably be faster to run it purely through a virtual machine, but I don’t really know if it’s possible to find a service somewhere (preferably with github integration) on which one could spin virtual machines and pull test results out of them ?
In general, I would be very grateful if the community points to literature on the topic.

Answer 1

So you formulate the question, what exactly does not work.
If the question is about automatic installation of an application - well, any application is ultimately just files and maybe a little bit of the registry.
You can take a clean virtual machine, turn on regmon / filemon and track what and where the installer puts. Then it will be possible to install it manually without the installer.
Of course, docker for Windows applications is possible, but docker is still more designed for the Linux area. In the case of the cloud, it is easier to make an image for a virtual machine from which to clone and run a test machine. But it depends on what you need.
But first of all, still more specifically formulate what you want

Answer 2

Start with an "algorithm" for manually installing and running tests, and then think about how to automate each step. The algorithm is usually divided into two parts: a one-time installation (you can do it manually and prepare an image of the machine), and what we do at each start.
I use Jenkins as my orchestration tool.

Answer 3

Apparently, you need to smoke xvfb to run the GUI without a real screen and without the mandatory binding to VNC