A
A
alexdora2016-06-27 16:01:50
MySQL
alexdora, 2016-06-27 16:01:50

Is this optimization worth doing?

Question to those who created highly loaded scripts on NODE.JS
Now 700 NODE.JS scripts are launched separately.
- Each creates one Mysql connection and approximately 2-3 SELECT / UPDATE queries to one innodb table
- Each creates a WS connection to a third-party service
i.e. 700 scripts / 700 Realtime Websockets connections / 700 database connections / and up to 2100 requests / sec to bd
There was a little time to work on optimization. I came up with the idea of ​​how to save the database from so many requests and connections => run one NODE.JS that creates an array of 700 connections to WS and, accordingly, only one connection to the database is needed. But there are big doubts that NODEJS will digest such a stream at all...
PS: I don't know JS well and it will take me a significant amount of time to create such a script with logic. I don’t want to just do it, and then it turns out that the script falls off.
PS2: I tried Redis to remove some of the queries or do without the database at all. But on my "nipple" system, I couldn't apply it correctly.
ADD:
I can't show the code. But everything is standard there: Connect WebSockets > infinite loop with a delay of 1 second with a request to the database (delay via Sync). If the database contains instructions new for a particular connection, ws.send is done.
Hold connection via ping/pong
ADD2:
Timur Shemsedinov: I kind of put everything together, but there were questions. If you can explain a little. I did not track (and it was difficult to track) how much memory all the scripts consume at once (But apparently not enough). This is normal behavior:
1. In the old scenario with 700 copies of one script, 700 to the database:
VPS / total RAM free 2gb out of 8gb / CPU [4 cores 12-14%] peaks
2. New scenario, one script, 700 connections WS, 1 connection and every second requests to the
VPS database / total 10.5gb out of 16gb (the script itself eats 4.5Gb) / CPU [4 cores 2%-3%] peaks
3. New alignment, one script, 350 (reduced by 2 times) WS connections, 1 connection and every second queries to the database.
VPS / total 10.3gb out of 16gb (the script itself consumes 4.4Gb) / CPU [4 cores 2%-3%] peaks
PS: In the last two cases, I added RAM VPS, because everything hung to hell at startup. But there is a lot of memory, this is not a problem.
I don't really understand how this happened and is this behavior normal for Node.js? XEN virtualization, Ubuntu 14.04, SSD
My asynchrony (if you can call it that) is done by Sync. WS and so asynchronously works. Sync performs a second restart:

function refreshDb()
  {
  Sync(function() 
      {
      Sync.sleep(1000);
      //Инструкции
                        refreshDb();
      });     
  }

Answer the question

In order to leave comments, you need to log in

1 answer(s)
T
Timur Shemsedinov, 2016-06-27
@MarcusAurelius

It makes sense to do 1 process, connection pool to a database of 20-30 connections, asynchronous code, and connection reuse, because one connection can execute one SQL at a time and will block the rest, you need to play around with the number of connections in the pool.
Useful links:
https://www.npmjs.com/package/mysql#pooling-connections
https://www.npmjs.com/package/mysql-utilities

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question