What can cause strange and high loadaverage in ubuntu?

Question

linux

On Ubuntu, over the past two weeks, the loadaverage has grown evenly, now it is 17.
The server has 16 cores, 132 gigabytes of RAM, a hard raid from ssd and no load, 100 gigabytes of free memory, 99% of the processors are idle, there is almost zero activity on the disks and a lot of free places.
I noticed a strange thing in the top from time to time one Apache and kworker jumps with cpu usage 9999, probably somewhere here lies the reason for how to diagnose such a problem,
Usually in the top you can immediately see who is eating the processor, but here the processors are free and it’s not clear where the queue to him.

Answer 1

wa doesn't grow at the same time that la grows by any chance ?

Answer 2

cat /proc/interrupts is interesting to see.

Answer 3

actually garbage, la \u003d 17 cores 16
17/16 \u003d 1.06
it turns out not so scary, but the system does not strain at all,

Answer 4

I heard about a similar problem recently. And no matter how strange it may sound, the kernel was to blame, which was "generic" and was constantly trying to get to the floppy disk, which was disabled in the BIOS. The problem was solved by simply disabling non-nuisance devices.

And for the first time I hear that load should be divided by the number of processors or cores! We have computers with 16 cores in production, and at load 45 I already receive notifications about a critical load, with such a load, the service is almost not available.

Answer 5

put htop, look at iostat, iptraf, sar. And so on the screenshot you can see that the Apache. :) Put nginx in front of it and I think the load will drop