Why does Sphinx crash every 15 minutes?

Question

linux

Hello, please tell me where to look and what can be done about it.
Version: Sphinx 2.2.11-id64-release (95ae9a6)

log

[Wed Oct 31 14:45:01.285 2018] [28649] watchdog: main process 28650 forked ok
[Wed Oct 31 14:45:01.286 2018] [28650] listening on all interfaces, port=9312
[Wed Oct 31 14:45:02.360 2018] [28650] accepting connections
[Wed Oct 31 15:00:01.918 2018] [28649] watchdog: main process 28650 killed cleanly with SIGKILL, shutting down
[Wed Oct 31 15:00:01.930 2018] [17088] watchdog: main process 17089 forked ok
[Wed Oct 31 15:00:01.931 2018] [17089] listening on all interfaces, port=9312
[Wed Oct 31 15:00:03.033 2018] [17089] accepting connections
[Wed Oct 31 15:15:02.104 2018] [17088] watchdog: main process 17089 killed cleanly with SIGKILL, shutting down
[Wed Oct 31 15:15:02.179 2018] [4889] watchdog: main process 4890 forked ok
[Wed Oct 31 15:15:02.180 2018] [4890] listening on all interfaces, port=9312
[Wed Oct 31 15:15:03.963 2018] [4890] accepting connections
[Wed Oct 31 15:30:01.338 2018] [4889] watchdog: main process 4890 killed cleanly with SIGKILL, shutting down
[Wed Oct 31 15:30:01.346 2018] [24725] watchdog: main process 24726 forked ok
[Wed Oct 31 15:30:01.347 2018] [24726] listening on all interfaces, port=9312
[Wed Oct 31 15:30:01.371 2018] [24740] watchdog: main process 24741 forked ok
[Wed Oct 31 15:30:01.372 2018] [24741] listening on all interfaces, port=9312
[Wed Oct 31 15:30:01.373 2018] [24741] WARNING: index 'base': lock: failed to lock /var/lib/sphinx/base.spl: Resource temporarily unavailable; NOT SERVING
[Wed Oct 31 15:30:02.534 2018] [24741] WARNING: index 'skidka_gid_aprods': lock: failed to lock /var/lib/sphinx/skidka_gid_aprods.spl: Resource temporarily unavailable; NOT SERVING
[Wed Oct 31 15:30:02.534 2018] [24741] FATAL: no valid indexes to serve
[Wed Oct 31 15:30:02.536 2018] [24726] accepting connections
[Wed Oct 31 15:30:02.569 2018] [24740] watchdog: main process 24741 exited cleanly (exit code 1), shutting down
[Wed Oct 31 15:31:31.686 2018] [26435] watchdog: main process 26436 forked ok
[Wed Oct 31 15:31:31.688 2018] [26436] listening on all interfaces, port=9312
[Wed Oct 31 15:31:31.688 2018] [26436] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:31:34.690 2018] [26436] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:31:37.693 2018] [26436] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:31:40.696 2018] [26436] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:31:43.700 2018] [26436] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:31:46.702 2018] [26436] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:31:49.705 2018] [26436] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:31:52.708 2018] [26436] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:31:55.711 2018] [26436] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:31:58.714 2018] [26436] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:32:01.717 2018] [26436] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:32:04.720 2018] [26436] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:32:07.723 2018] [26436] FATAL: bind() failed on 0.0.0.0: Address already in use
[Wed Oct 31 15:32:07.725 2018] [26435] watchdog: main process 26436 exited cleanly (exit code 1), shutting down
[Wed Oct 31 15:45:02.287 2018] [10730] watchdog: main process 10731 forked ok
[Wed Oct 31 15:45:02.290 2018] [10731] listening on all interfaces, port=9312
[Wed Oct 31 15:45:02.290 2018] [10731] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:45:05.290 2018] [10731] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:45:08.291 2018] [10731] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:45:11.294 2018] [10731] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:45:14.297 2018] [10731] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:45:17.300 2018] [10731] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:45:20.303 2018] [10731] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:45:23.305 2018] [10731] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:45:26.308 2018] [10731] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:45:29.311 2018] [10731] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:45:32.314 2018] [10731] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:45:35.317 2018] [10731] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 15:45:38.320 2018] [10731] FATAL: bind() failed on 0.0.0.0: Address already in use
[Wed Oct 31 15:45:38.327 2018] [10730] watchdog: main process 10731 exited cleanly (exit code 1), shutting down
[Wed Oct 31 16:00:01.469 2018] [28236] watchdog: main process 28237 forked ok
[Wed Oct 31 16:00:01.471 2018] [28237] listening on all interfaces, port=9312
[Wed Oct 31 16:00:01.471 2018] [28237] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:00:04.474 2018] [28237] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:00:07.476 2018] [28237] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:00:10.479 2018] [28237] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:00:13.482 2018] [28237] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:00:16.484 2018] [28237] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:00:19.487 2018] [28237] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:00:22.490 2018] [28237] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:00:25.493 2018] [28237] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:00:28.496 2018] [28237] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:00:31.499 2018] [28237] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:00:34.502 2018] [28237] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:00:37.505 2018] [28237] FATAL: bind() failed on 0.0.0.0: Address already in use
[Wed Oct 31 16:00:37.514 2018] [28236] watchdog: main process 28237 exited cleanly (exit code 1), shutting down
[Wed Oct 31 16:15:02.598 2018] [13212] watchdog: main process 13213 forked ok
[Wed Oct 31 16:15:02.604 2018] [13213] listening on all interfaces, port=9312
[Wed Oct 31 16:15:02.604 2018] [13213] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:15:05.607 2018] [13213] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:15:08.610 2018] [13213] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:15:11.613 2018] [13213] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:15:14.616 2018] [13213] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:15:17.619 2018] [13213] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:15:20.621 2018] [13213] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:15:23.624 2018] [13213] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:15:26.627 2018] [13213] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:15:29.630 2018] [13213] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:15:32.633 2018] [13213] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:15:35.636 2018] [13213] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:15:38.639 2018] [13213] FATAL: bind() failed on 0.0.0.0: Address already in use
[Wed Oct 31 16:15:38.649 2018] [13212] watchdog: main process 13213 exited cleanly (exit code 1), shutting down
[Wed Oct 31 16:30:01.479 2018] [31275] watchdog: main process 31276 forked ok
[Wed Oct 31 16:30:01.481 2018] [31276] listening on all interfaces, port=9312
[Wed Oct 31 16:30:01.481 2018] [31276] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:30:04.482 2018] [31276] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:30:07.485 2018] [31276] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:30:10.487 2018] [31276] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:30:13.488 2018] [31276] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:30:16.491 2018] [31276] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:30:19.494 2018] [31276] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:30:22.497 2018] [31276] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:30:25.499 2018] [31276] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:30:28.502 2018] [31276] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:30:31.504 2018] [31276] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:30:34.507 2018] [31276] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:30:37.510 2018] [31276] FATAL: bind() failed on 0.0.0.0: Address already in use
[Wed Oct 31 16:30:37.544 2018] [31275] watchdog: main process 31276 exited cleanly (exit code 1), shutting down
[Wed Oct 31 16:45:01.834 2018] [14835] watchdog: main process 14836 forked ok
[Wed Oct 31 16:45:01.836 2018] [14836] listening on all interfaces, port=9312
[Wed Oct 31 16:45:01.836 2018] [14836] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:45:04.838 2018] [14836] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:45:07.841 2018] [14836] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:45:10.844 2018] [14836] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:45:13.847 2018] [14836] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:45:16.851 2018] [14836] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:45:19.852 2018] [14836] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:45:22.855 2018] [14836] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:45:25.858 2018] [14836] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:45:28.861 2018] [14836] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:45:31.864 2018] [14836] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:45:34.867 2018] [14836] bind() failed on 0.0.0.0, retrying...
[Wed Oct 31 16:45:37.868 2018] [14836] FATAL: bind() failed on 0.0.0.0: Address already in use
[Wed Oct 31 16:45:37.876 2018] [14835] watchdog: main process 14836 exited cleanly (exit code 1), shutting down

Answer 1

Because:
What is not clear in this message?

Answer 2

This is what happens when a process runs out of memory. When you try to reserve a piece of memory 56 GB by the Sphinx !! got a bummer and his process was killed with the help of oomkiller - a standard Linux mechanism.
Perhaps every 15 minutes some hellish request to the Sphinx should be performed (for example, because of the bots) and this is how this result is obtained.

Answer 3

First, you send him a signal to die constantly, then he sees a busy port, which means another sphinx is already sitting on the port