How to make a virtual machine security system?

Question

linux

Good afternoon!
It so happened that I got the thesis topic related to virtual machines. The bottom line is that you need to make a system that will allow you to control the user's actions in the guest OS, for example, you need to find out what he launched, what he downloaded, and in which case, take the necessary actions, for example, throw out alerts or prohibit any actions. In short, such an antivirus is at the hypervisor level. I know that these things already exist, but I need my own bikes and crutches.
All this should work at least on ESXi and work without an agent on the guest OS. Actually, the initial task is exclusively monitoring the system.
Personally, I see 3 ways how to do this:
1) With the help of vshield endpoint api - in fact, with the help of this API, many industrial solutions already work. But I did not quite understand how you can use this thing for monitoring.
2) Nested virtualization - write something like a blue pill. Run this thing in the hypervisor, run another hypervisor inside and actually profit on the face. I myself believe that this is the most suitable solution.
3) Deal with vd-x and vd-t. Pick it up in some way, well, and there it will be seen what can be done.
I wanted to ask what other options are there, where to look, where to dig, what to read?

Answer 1

A brief educational program on how virtualization works, in order to advance you a little bit on the issue of a solution.
Very enlarged and simplified, it works in cycles. Each cycle, the processor performs some actions, considers something, reads something, writes something.
Each cycle, the processor receives a set of instructions and they are executed.
In each cycle, we have a piece of memory with which we can work - registers. Some we can read and write, some we can only read, some show us the current state. There are many registers and they are different.
We have an interrupt table - it shows us the state of the system, the state of the hardware and the input and output devices.
And a lot more. All this is calledExecution context .
Among other things, the processor (again, to simplify) has a security system based on process privilege levels. For Windows (and many other systems), for example, these are two levels: kernel (zero level, kernel level) and user (User-mode). A process (in this approximation, instructions) running at different levels have different privileges, different access to registers and other resources, and have different execution context
The operating system starts applications, transfers control to them for a certain time, divides the processor between applications according to their priorities, their needs, the focus on the window, interruption from hardware or drivers, takes the processor if there is a processis more important or if there is an expectation of something (reading, writing disk, network and tydy).
The OS itself and its drivers, processes, services, and so on - work in a certain Execution Context , it partially changes for each application, but not significantly.
The OS (its kernel), part of the processes and part of the drivers - work at the zero level, with maximum privileges, can write / read pieces of memory, write to any place, can read most registers, and so on.
Nothing. Well, that is, just like that, for the OS there is no process or view, for the OS (host OS) there is no virtual machine.
There is a process - a hyperwizard (if a little more precisely - virtual machine monitoring - VMM). Its tasks include monitoring the execution context, interrupts, and taking control from the guest or host OS. Change the execution context and return everything back.
OS, sighing, sitting down on the couch:
- It seems I'm a little tired and there is no more work for today
VMM:
- Then perhaps I should go.
He leaves the room with the teapot
VMM:
- Psst, listen, are you interested in the processor?
Someone:
Oh! And I like the mask of the processor and it's generally cool, only I don't see the registers, where, let VT-D?
VMM:
- What did you say? Will you take no?
Someone:
- Come on, okay, I just asked, I will of course teach me how to be a virtual machine
The room is smoky, bottles are scattered, it smells of burnt rags. Somewhere they laugh.
A man in shorts and with a PS4 remote control stared at the monitor. The door breaks off its hinges and falls to the floor.
VMM bursting in:
- Everyone get out of the dusk! It's the night watch!
*translating into computer terms.
For VMM OS, this is a process. The OS passes control to it like a normal process. The one that changes the execution context (changes all all registers. Or almost all all registers) and transfers control to the guest OS, which does some of its work (knowing that it is virtualized). Next, the hyperwizard pauses the guest OS and returns everything back. After that, the host OS gets its execution context and does its job.
And so beat after beat.
Did you get an interrupt? We checked to whom it was, changed the execution context, delivered an interrupt, cleaned the tables, and returned it back.
**Actually, it's a bit more complicated. And with interrupts, and with access levels, and with registers, and with equipment, and tydy, but this is beyond the scope of the question (and I'm too lazy).
*** We are talking about hardware virtualization.
Returning to your question. How do you want to see from the host system what is happening in the guest OS without "special intervention"?
Change the theme of the thesis. Dig into how virtualization works. This in itself is a very good topic for study.

Answer 2

Well, what if I run net/freebsd, windows, dos, os/2 under a virtual machine, God forgive me? Alas and ah, it is possible to control a virtual machine only indirectly, if we do not have a complete idea of ​​​​what the guest was running:
- by the file system and / or disk exchange, and even then, if the guest does not use cryptography on the disk;
- network traffic, and again, if the guest does not encrypt it, or tor does not start there.
- of course, you can use vd-x and vd-t to look at the contents of registers and memory, catch any I / O.
But it all starts to resemble a game of code reengineering with obtaining access keys to some kind of paid application. It seems that the application is running in our system, it only requires keys for its launch. It seems to be like our computer, and we are not the owners of the application. And debuggers seem to be able to launch and change all sorts of registers, only hell is there ...
Something I don’t like this topic of the diploma ...
PS. The other day I was running Windows NT4.0 SP3 in a virtual machine. It started, it's a sad activity, I'll tell you ... There are no drivers, forwarding old PCI-boards is a horror. The new ones do not work, there are no drivers, and the software does not support them. NT itself is still that girl with pigtails from Hitchcock films ...

Answer 3

agentless monitoring is not possible. To know what is happening in the virtual machine, you need to have "your person" there, otherwise you look like a person who stands in front of a closed door and tries to figure out what they are doing in the room based on the noise behind it, as well as counting the incoming and outgoing people.. Watch the
Soviet film "The Bartender from the Golden Anchor". The film itself is mediocre. But it has an idea - a whole spy operation is turned for the sake of installing an agent (sensor)!