How to properly install prometheus to collect metrics for k8s and other systems?

Question

Monitoring

We have a couple of internal clusters with k8s without ingress with a VIP address, and a bunch of separate VMs with various services on windows and linux.
There was a task to collect metrics from these applications. The choice fell on Prometheus. But not everyone lives in k8s, some things are difficult to transfer.
Google on this subject made me stupor, for k8s they mostly suggest using prometheus-operator.
https://github.com/prometheus-operator/prometheus-...
I looked at their bundle.yml and got it) 1Mb of code. At the same time, I need to somehow deploy configs for other services outside of k8s.

In some articles, they suggest installing through helm.
https://habr.com/ru/company/agima/blog/524654/

Which use case would be better in my case, so that configs for other services are stored in git?

Answer 1

I looked at prometheus-operator from coreos, they also have helm in their options, but the name has changed.
on kube-prometheus-stack. In fact, this is a set of services integrated with each other and tailored for k8s.
I made it so that for servers with k8s, the stack was raised via kube-prometheus-stack, but without alertmanager and grafana. And for the rest of the services I made a separate VM and raised the usual prometheus, grafana, alertmgr there and through federation I set up the collection of metrics from k8s clusters.