Why does the speed of math operations in C++ change?

Question

C++ / C#

There is a two-dimensional (also with three-dimensional) dynamic array. Data is loaded into it and then it is used. The data weighs several GB. With an array, simple mathematical operations are performed - addition, multiplication, root, and the like. (The result is not saved to the array. That is, the array is permanent.) Everything happens on the Intel Xeon E5-2660 v4 in 10 threads. (The processor itself is 14 cores). The question is why once the same operations go - 0.1 sec. Another time (if you do nothing for 2 minutes) - 1 sec. Why is this happening? Maybe you need to change something in the settings of Visual Studio 2015 Release? Or in the server settings? OS - Windows Server 2012 R2.
PS No other operations slow down. It's the mathematical ones. Or access to an array (this may be)?
PPS I also noticed that when operations are performed normally (for 0.1 seconds), the processor is loaded by 10-15%. When in 1 second - more than 40%.
No other operations are performed on the server. So the processor is free to the maximum. RAM is also free enough (several GB).