E
E
Egor Petrov2016-03-27 11:48:49
Java
Egor Petrov, 2016-03-27 11:48:49

How to cure ActiveMQ from freezing and understand the reason?

The problem is this, ActiveMQ had a wonderful uptime, after the last change in settings, two months. Slowly, without radical jumps, the load grew. And there were a couple of incidents over the weekend.
ActiveMQ hangs, almost all 8 cores of the processor survive, the web console does not open, consumers cannot connect.
For the first time, the ActiveMQ logs contained the following entries:

2016-03-26 02:16:19,552 | ERROR | Could not accept connection : java.lang.Exception: java.lang.OutOfMemoryError: Java heap space | org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server Thread Handler: stomp://127.0.0.1:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2016-03-26 02:17:37,002 | ERROR | Could not accept connection : java.lang.Exception: java.lang.OutOfMemoryError: Java heap space | org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server Thread Handler: stomp://127.0.0.1:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2016-03-26 02:17:37,794 | WARN | MBean org.apache.activemq.broker.jmx.ManagedRegionBroker | ActiveMQ Transport: tcp:///127.0.0.1:[email protected]
2016-03-26 02:18:30,999 | ERROR | Could not accept connection : java.lang.Exception: java.lang.OutOfMemoryError: Java heap space | org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server Thread Handler: stomp://127.0.0.1:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2016-03-26 02:18:37,485 | WARN | Exception occurred processing:
CONNECT
host:127.0.0.1
accept-version:1.0,1.1,1.2
passcode:*****
login:login
: java.lang.OutOfMemoryError: Java heap space | org.apache.activemq.transport.stomp.ProtocolConverter | ActiveMQ Transport: tcp:///127.0.0.1:[email protected]
2016-03-26 02:21:10,769 | ERROR | Could not accept connection : java.lang.Exception: java.lang.OutOfMemoryError: Java heap space | org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server Thread Handler: stomp://127.0.0.1:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2016-03-26 02:21:10,769 | WARN | Transport Connection to: tcp://127.0.0.1:32482 failed: java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java heap space | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///127.0.0.1:[email protected]
2016-03-26 02:21:50,985 | ERROR | Could not accept connection : java.lang.Exception: java.lang.OutOfMemoryError: Java heap space | org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server Thread Handler: stomp://127.0.0.1:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600

The second time, nothing unusual was written, the last thing before the freeze was the broker's launch logs.
To work with ActiveMQ, consumers and producers in PHP are used, the work goes through stomp. Messages are both persistent and regular. Large volumes of messages in queues do not accumulate, they are immediately sorted out by consumers, more than 30 messages at one time in the queue cannot be seen. (In general, earlier, when there was a case of a consumer crash, several hundred thousand messages accumulated in the queue and nothing fell, in the morning the consumer was launched, the messages were parsed and everything went without consequences. And another time, I remember, there were more than a million messages. )
ActiveMQ version 5.10.1, everything works with openjdk
java -version
openjdk version "1.7.0_80"
OpenJDK Runtime Environment (build 1.7.0_80-b15)
OpenJDK 64-Bit Server VM (build 24.80-b11, mixed mode)

OS: FreeBSD 10.1-RELEASE-p16
At the moment, I don't even quite understand what I need to monitor in order to at least understand the cause of the problems. The only thing I googled is that ActiveMQ behaves like this if all it does now is garbage collection, that is, almost 100% of the processor resources are consumed by the GC. Not the fact that this is so, and I see no reason for this.
Has anyone experienced something similar?

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question