How to monitor the state of the application?

Question

Monitoring

There is a server application in the form of a demon. The task is to monitor its state and see the consumption values ​​of some resources (including some internal resources, the value of which can only be obtained from within the application). How best to implement?

Options come to mind:
1. Zabbix or similar monitoring system. In this case, do I need to write a plugin for the system, or is it better to use, for example, SNMP? The plugin meets all the requirements, except that it is not universal, I would like it not to be tied to a specific monitoring system. SNMP - authentication immediately scares away, or rather its absence (I'm not special in administration, I judge by zabbix, it just enters the address of the SNMP agent and that's it, no login passwords) - this is not acceptable. I also noticed IPMI in Zabbix, but this is more for setting up the platform, it will probably look strange to monitor the application. JXM is not suitable - not Java.
2. Monitoring via web. There is no problem, except that it is useless. Yes, it will be possible to look at changing numbers and beautiful charts, but we need a normal monitoring system, something unified that supports automation and is integrated with other monitoring subjects.
3. Write your own softinka. The same problem as in point 2.
4. Well, I don't know. Maybe use some third party service?

I would be most satisfied with the implementation of some standard protocol (for example, SNMP).

What do you think about this? If it is difficult to give an unambiguous answer, then I would be grateful if you would evaluate / compare existing solutions. In general, how do you solve this problem? (Well, it can’t be that all monitoring is limited only to monitoring Apache / nginx / mysql ?!)

Answer 1

Zabbix can execute any commands in the agent. You can add some simple CLI interface to your system so that bin/status returns plain text data or searches by some parameter.
I have it implemented so that applications store their statuses and key indicators in Redis, and Zabbix reads from there. It turns out that from the side of zabbix, you only need to know the key, from the side of applications, radish is already used everywhere

Answer 2

Of course, the choice is Zabbix (or commercial monitoring systems).
Not 2 by the fact that for monitoring it is often necessary to increase the access level (some points can only be recognized by the root, or someone from the XXX group). Doing this through the web is dangerous.
Not 3. By the fact that why write something that was written a long time ago and is actively used.
Not 4. By the fact that not everything you can give out. And I do not trust these proxy agents.
SNMP - authentication immediately scares away, or rather its absence- everything is fine with zabbix snmpv3 authpriv authentication. There is even encryption. But besides SNMP, any transport or plugin can be attached to Zabix. For example, he asks me about the state of bases and fronts via WEB. In enterprises, monitoring is almost always done via SNMP. There are even pieces of iron that wrap the serial port in SNMP;)
In short, read in groups. Zabiksovoe community can monitor almost everything.