Application monitoring in Openshift 3.11. Do you need Prometheus for your services?

Question

Monitoring

I welcome everyone.
Please provide an answer to this question.
There is Openshift 3.11 configured on a turnkey basis by the contractor. A bunch is already configured on it:

.\oc.exe -n openshift-monitoring get routes
NAME
alertmanager-main
grafana
prometheus-k8s

to monitor Openshift itself.

Actually the question itself - do I set up a new instance of Prometheus for my applications and services, or use an existing one? Is it possible to use the existing one in principle for your services . Since there is https://docs.openshift.com/container-platform/4.6/... but this is the 4th version, and there is no such document for the 3rd one at all.

Answer 1

I have very little experience in this matter, but still: why set up a new instance of Prometheus? You have a cluster in which Prometheus is installed and which scrapes the targets that you specified to it in the config.
As an option, you can consider the option with the federation. This is if you need, for example, to store metrics for a long time, then you can have an instance of Prometheus inside the openshift, and put another instance of Prometheus outside, which will scrape that Prometheus from openshift and store the metrics for a long time (for example, --storage.tsdb.retention.time=365d).
In general, there is such a company Flant, they have a blog on Habré and on YouTube, they have a good video about monitoring inside the cube.

Answer 2

Of course, it is better to use a ready-made copy, there would be access to Prometheus.
As a last resort, you can make your own exporter for custom metrics, on some python, which will get the metrics the way you came up with (json via http endpoint, or parse logs or something else, up to side-car inside each pod ) and throw a general Prometheus base.