Lots of mysql connections. Can't connect to MySQL server on '192.168.1.1' (99) how to requeue?

Question

MySQL

Good evening, we have a highly loaded project
that is distributed over 2 servers.
1st server (centos 6): 8 core 3400MHZ. 32 gig memory.
On the 1st server there are php site scripts, css, js and so on.
Build: PHP 5.3.3 (cli) (built: Jul 9 2015 17:39:00)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies
with XCache v3. 2.0, Copyright (c) 2005-2014, by mOo
with XCache Cacher v3.2.0, Copyright (c) 2005-2014, by mOo
nginx version: nginx/1.8.0
php built as php-fpm.
2nd server (debian 7): 12 core 3800MHZ. 64 gig memory.
On the 2nd server we use mysql 5.5.46
Both servers stand on 1 switch and are connected to each other in one network.
From the 1st server we connect to mysql via internal ip: 192.168.1.1 we connect
to the database via mysql_connect
The script engine of the site is designed so that we open the connection require_once './config.php';
from below close mysql_close($con);
We have a problem because of the large number of connections, sometimes, especially when more than 4000 people are online, it gives out:
Can't connect to MySQL server on '192.168.1.1' (99)
During the online period on the site of 4000-5500 people, there is no load on both servers :

At the moment, perhaps our problem would be solved by switching to persistent connections mysql_pconnect(),
but we can't connect to mysql database using mysql_pconnect() because we have php built as php-fpm and not apache module.
All heavy queries that were running for a long time were tracked using long_query_time and optimized!
The mysql database is currently accepting: ~4,000 requests per second. ~ 360,000,000 requests per day.
[email protected] ~ # hdparm -t /dev/sda
/dev/sda:
Timing buffered disk reads: 1418 MB in 3.00 seconds = 472.50 MB/sec
Of course not ssd, but the result is very good, the speed is very close to ssd.
In this case, since we have a large number of connections to mysql, do we need to build up a queue for connections to mysql? But increasing back_log doesn't help.
How can we set up a queue for connecting to the mysql database so that connections do not fall off and issue Can't connect to MySQL server on '192.168.1.1' (99), but wait for their turn?
All mysql settings not skipping my.cnf settings

Answer 1

> Timing buffered disk reads
crap =) Compare ram+hdd and ssd.
perror 99 = ran out of src ports, according to perror.
You need to watch what happens in netstat -tunlp at this time.
> rand(3306,3308)
won't help. You ran out of src ports, not mysql.
If everything is really bad (well, that is, nestat will not prompt thoughts and it will not work to switch to keepalive), you need to make several ip-addresses on both nodes, configure routing from the src-address on the first to the corresponding address of the second (this will be the cheapest in terms of resources ) and randomize already IP.

Answer 2

Try it, maybe habrahabr.ru/post/129482
update will help: And probably you should think about switching the project to at least php 5.6 and changing mysqli to PDO. If the project grows, then it will get worse. It is also worth thinking about using nginx cache for fpm, with its help we removed the load from the backend and increased the response speed from the server from 550 ms to 180, given that the servers are in the USA.