Xen + mysql = high load average on low CPU load?

Question

MySQL

Good day, habr. I ran into such a problem - there is a machine (2x Intel Xeon E5410 + 8 gb ram) on Debian 6 (x86_64) + Xen 4. domU is spinning on it (6 cores and 3 GB of RAM, for a database with a size of about 5 GB). Initially, this vds had 4 cores and 2 GB of RAM, but I noticed an oddity - the processors are evenly loaded by about 30-40%, Load average reaches 20, mysql slows down, the programmer is indignant. The situation was saved by restarting mysql, because while the base was being tinkered for itself, it was not scary. It seemed to me that there were not enough processors, I added 2 more cores and 1 GB of RAM, up to 6 cores and 3 GB of RAM. LA can now go up to 30, mysql slows down again. However, the processor load is lower, in the region of 15-20%. I poked top - the problem is not in the disk, not in the processors, not even in the RAM. I was confused by the high value of the %st column, which reaches 85%.

Answer 1

So, problem solved. To whom it is interesting - I will describe. In general, top and atop on the host machine (dom0) showed 99-100% idle cpu, the disk was used at 5%. Inside VDS (domU) wild brakes. They gave me a link - CreditScheduler . xm sched-credit gave me weight 86 (86% of 1 core). Those. 6 virtual cores accounted for less than 1 physical. Fixed viaxm sched-credit -d database -c 0 , where database is the name of the wdski. I set 600, the processors loaded evenly. top in dom0 still reports that the host machine is not loaded, but the joke is that top is lying - the load is noticeable through small console brakes and through xentop, which shows how much processor time each WDS eats. Well, a small optimization of a couple of bold queries - now queries are executed many times faster in VDS, and VDS has honest 6 cores, which are executed on 6 real cores, and not on 0.86 of one core.
Hope it helps someone.

Answer 2

This is steal time - the percentage of CPU time that the host used on its own while your virtual machine needed it.