How to write back-end, which has a lot of mathematics?

dom1n1k2015-11-16 22:28:34

Python

dom1n1k, 2015-11-16 22:28:34

There is an idea for one service whose backend will calculate quite a lot of mathematics on very voluminous data (statistics, fft, convolutions, sound and image processing, maybe a little recognition and all that). Actually the question is which language to choose?
Some time ago I learned about NumPy and really liked it. In this regard, of course, the idea arose to choose Python. One experienced friend added fuel to the fire, who said something like "yes, now the whole web needs to be done in Python" (I will mentally make allowances for his personal sympathies).
But then I came across an article , from which it follows that Python is on average an order of magnitude (!) Slower than Go. This greatly shattered my opinion. 2-3 times could be forgiven, but order is a very serious reason to think. Servers are not cheap.
And so I thought, what can I do differently - the main part of the backend is stupidly in PHP (easier and cheaper), and the most critical and mathematical places in Go?
On the other hand, I read that NumPy seems to be internally implemented in C ++, and python is only an interface there? If so, what tests in the above article might not be indicative in this case?
And so as not to get up twice - are there libraries like NumPy for PHP, Go, etc.? To have a similar ideology, the number of functions and so on. Of course, something native in C ++ (like FFTW) will work the fastest, but I don’t want to bother with a large zoo of disparate libraries in an attempt to make them friends. I would like a swiss knife.

Answer the question

In order to leave comments, you need to log in

10 answer(s)

nirvimel, 2015-11-17
@nirvimel

Python is slow for computing!? You just don't know how to cook it!
I already wrote here that python is only a dynamically typed script from above (which is necessary for development speed), but numpy vector calculations are performed on the hardware itself, that is, so that you do not write it in C / C ++ faster than a few percent.
In addition, for those cases when vector calculations are not enough, there is Cython, it is the same compiled (and not inferior in performance) as C / C ++ language, with direct access to the python object passed from the script.
All this is implemented through vector calculations or through the corresponding libraries, which are also not written in a script and which will not yield to the ideal solution by more than a few percent in performance. Even if for some particular task there is no ready-made solution in the form of a library, then there is always the option to make this little thing in Cython.

Arman, 2015-11-16
@Arik

Write everything on the faster it will turn out, look at the server load, which parts of the software are loading and quickly fix it. And so tomorrow you will read that the assembler, or something else, does all the languages \u200b\u200bcombined and start rewriting again. And so you at least have a working option which is no and in a short time. It may not be necessary to rewrite anything, this often happens to me.

s0ci0pat, 2015-11-16
@s0ci0pat

In C++, and preferably in C. We do just that.

lega, 2015-11-16
@lega

I didn’t read the article, I would take python for everything (web) and c ++ for mathematics.
Go for network problems, in mathematics* it is inferior to c++

best_santa, 2015-11-16
@best_santa

C++ for bottlenecks, everything else, whatever is more convenient.

Oleg Tsilyurik, 2015-11-16
@Olej

from which it follows that Python is on average an order of magnitude(!) slower than Go.

Well, the rumors are greatly exaggerated. ;-)
Actually there are 2 orders (Python2) or more (Python3).
Someone suggested here:
"We do it this way" is certainly a strong argument... but Go is not slower than C/C++,... well, on some types of tasks it is up to 2 times slower.

Roman Mirilaczvili, 2015-11-17
@2ord

The subject area of digital signal processing is best written in high-performance languages, collected in libraries. Then from any web framework, say Python, you can work with library calls.
In general, Fortran / Go / Java are suitable for the back-end. Fortran is mostly written by bearded guys. And Go/Java is for the current generation.
As an experiment, I can also offer the Julia language , which, according to the developers, has a speed close to C. Moreover, this language under the hood uses calls to mathematical libraries: FFTW, LAPACK, OpenBLAS, GMP, etc.

un1t, 2015-11-17
@un1t

Indeed, if you use numpy or other libraries written in Xia, then you won’t be able to write faster in Go. The question is how much of your logic fits into these sish libs.
Here the guys write that now you can write modules for python on Go!
stackoverflow.com/questions/12443203/writing-a-pyt
... you can rewrite slow sections of code in Go.

Mikhail Potanin, 2015-11-26
@potan

I recommend watching Julia . The language is comparable to matlab in terms of mathematics, and more powerful than python as a general purpose.

Alexey Akulovich, 2015-11-26
@AterCattus

If Go is just one of the options here (and you don't know the language), don't choose it. If you know and like it, take it as a great solution for implementing backend daemons, and the most cpu-bound things through cgo, implemented in C ++.
Although here I have a little thing on Go that counts just FFT and about that, so the speed of Go itself was quite enough for me. The only question is, what is your load and requirements: +/- 2ms per request did not play a role for me in this case.