How to properly implement a distributed computing system?

Question

Distributed Computing

I have a rather unusual task... I
have a project (node.js) that is quite heavily loaded.
Project objective:

• The database (Redis) has over 20 million large records (each has over 100 different parameters in a JSON string)
  
• There is a very complex function that modifies a record.
  
• Each entry must be processed by the function once every 10-15 seconds, and produce a large object as a result (specifically, a huge log of its work)
  
• Allowable delay no more than 10 seconds (a new iteration should not be scheduled if the old one has not been completed, otherwise the system will crash)
  

Available resources:

• 5000$ for iron
  
• about 100.000 concurrent clients (Android(Unity 3D), iOS(Unity 3D), browsers(JS, support since IE10)) on WebSocket
  

As you can see from the task and resources, the $5,000 server budget is not designed for such an application, but we have quite a few active clients. Therefore, the idea of ​​​​distributed computing by clients suggests itself.
The scheme is now as follows:
1. All records during each iteration are placed on the FIFO stack, let's call it the "task stack"
2. A handler function is sent to each client during the start of the session (it changes often, we cannot allow it to be implemented in the client code )
3. Each client, when he has nothing to do, takes a task from the stack. Executes it and sends the result to the server.
4. The server puts the result in the FIFO stack, which we will call the "validation stack"
5. Each client also periodically takes on the task of validating results that are not their own (we cannot trust one client too much). Performs validation and sends the result to the server.
5.1. If the validation is successful, we modify the record and close the task.
5.2. If the validation fails, the server itself will personally deal with the task
. But the problems are as follows:

• There are no problems with browsers - there clients on JS will cope with the function quite well, but we have too few browser clients (no more than 10,000), mostly they sit from phones and tablets. Is it possible to organize a cross-browser function that works both on the server and on all types of clients in the same way? If yes, then in what way?
  
• Is it even possible to do this? If so, is it necessary to warn users? We do not plan to heavily load clients, we will take short breaks between calculations, otherwise we will lose confidence in the application due to reviews like "eats battery in a few hours." We also do not plan to load clients whose application is minimized (only in the case of mobile platforms).
  

UPD:
The server function execution time is about 30ms
UPD 2:
I should probably briefly describe the functionality:
Records in the database are the characters of the players.
The handler function is the generation of a fight between two characters. The result is a complete combat log, which is later played back by the client.
Only the data that is needed and nothing more are passed to the input of the function.
The function is quite well optimized, but due to the complex mechanics of the game, this function is still difficult (imagine a fast match generator between bots in a game like Heroes of Might and Magic in battle mode , triggered two million times per second).
Just in case, I will also clarify that the rest of the application is also optimized. For example, in real time we count only the characters of online players... And the battles for the rest are counted once per hour/day/week (depending on the date of the last activity), and we consider not all battles as such players, but only a part of them, increasing reward multiplier.
UPD 3:
The way to solve the problem by changing the language and buying specific hardware is of course good, we should have written a project in Erlang at all ... because we didn’t know that there would be such a problem, and considered node.js to be a suitable language that could go through such stress- test.
But let's still try to find answers to the questions:

1. About the ability to effectively execute a function written in one programming language (absolutely any high-level language, not necessarily JS, maybe even Lua), different programming languages.
   
2. On the legality of performing such calculations on the client, with or without the knowledge of the user. After all, these calculations are performed by the user not for himself, but for a third party.
   

Answer 1

Самый лучший выход на мой взгляд сделать эту возможность как задание в игре. То есть клиент сам выбирает активировать ее или нет и насколько сильно можно загружать проц и взамен за рассчет кадой игры получает какие то внутриигровые бублики, по сути они работают на вас, ведь с таким же успехом вы могли бы майнить битки на их видюхах, вместо рассчета игр, so they have to be paid for it, in proportion to the work they do, and then the players will be happy to do it. naturally, it is not necessary to tell them exactly about the calculation of games, but to beat them with gameplay (for example, accelerated learning of magic with x2 speed requires additional computing power for the player + explain to the player that the more powerful the computer, the faster the study will go). + it is possible to make an option to activate this thing only when powered by the mains, so many players will intentionally leave the phone on while charging just to get bonuses.

Answer 2

It is difficult for you to help even if you want. First of all, you need to see what the line with the parameters is, whether everything is needed in it. Let's all. Is everything needed right now? You probably saw that the statistics (percentage of wins) at WOT is updated once a day at night - it's no coincidence. Pavel
rightly said about GO , look at the Ogame sources on the net - the whole game is in php there, but the combat system is written in one of the C-languages. Why do you need a distributed system? Buy 1 server for $5000 with almost no RAM and HDD, order more processors there and write a handler in GO, C++, etc. Send all tasks according to calculations to it. PROFIT! )))

Answer 3

I'm not sure if this is possible in your system, but you can try to introduce a system of "time stamps", while "inactive" objects are not calculated by your function without special need, "freeze" them until they are accessed, when they were accessed calculate the function as many times as necessary, so that the "time" in the object "catch up" with the objective, while free computing power (when there are few clients) to spend on the calculation of "lagging" objects.