What is the best language for a metasearch engine?

Question

PHP

I want to try to do a metasearch, something like aviasales. I know php well, but for some reason, it seems that it is not very suitable for these purposes. Causes:

• Such a search implies the simultaneous polling of a large number of partners, and php in asynchrony, or multithreading is not very good.
  
• As I understand it, after the start of the search, the script returns http 200 and goes to poll partners in the background, then puts the results in some storage and another script pulls these results from there. Accordingly, if many people search at the same time, then php processes will multiply in memory that live longer than the request, and this can quickly put the server down.
  

Of course, I would like to do it in a language that I know, but how correct are my conclusions? Is it suitable for this or should I look towards something else? Maybe there are some other tips?

Answer 1

The language only allows you to implement the architecture. Even in languages ​​that are more asynchronous, with poor architecture, you will get the same resource exhaustion while you are talking in terms of a single script. And taking into account the ignorance of the features of this new language, you can lose a lot of time on debugging.
Therefore, the output is in architecture. Asynchrony is implemented through queues, and load reduction through caching.
A certain balance can be maintained by using different languages ​​through the queues in those places where they are needed.
Example:
1. API point where the results of parallel requests are collected - Node.js with promises that work with queues for communication with clauses 2 and 3, if necessary, rewrite in golang in the future if it slows down;
2. Points where high loads and a lot of logic are not needed - php, python, any language that you know well. If something becomes a bottleneck - rewrite compiled and optimize;
3. Where the load is high - something compiled like golang.
Accordingly, caching at the endpoint level allows you not to make new requests each time and reuse the results of the previous ones, and therefore the load will grow more smoothly.