Answer the question
In order to leave comments, you need to log in
Kubernetes: unable to add custom service-monitor?
Good afternoon.
I currently have prometheus installed using Helm3 from here.
It seems to work, taking metrics from my pods, which are all in the same node.
The question is that my service at :8085/posts/metrics is already taking metrics in the right format for Prometheus. An exporter is not needed. Accordingly, as far as I understand, I need to add a new service-monitor with the specified metrics reading address.
Unfortunately, I can't find any information on how to do it correctly.
From what I found is this post on stackoverflow:
As far as I understand, I need to add a service first, and then write a separate yaml file for service-monitor so that prometheus and grafana can see them, or can I add information to an existing service that takes metrics, and add service-monitor separately?
I tried to do it by analogy, but somewhere I write something wrong, because does not see prometheus
# Service targeting gitlab instances
apiVersion: v1
kind: Service
metadata:
name: http-metrics
labels:
app: http-metrics
spec:
ports:
- name: metrics
port: 8085
endpoints: /posts
targetPort: metrics
protocol: TCP
selector:
app: http-metrics
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: http-metrics-servicemonitor
# Change this to the namespace the Prometheus instance is running in
# namespace: default
labels:
app: http-metrics
release: stable
spec:
selector:
matchLabels:
app: http-metrics
endpoints:
- port: metrics
interval: 15s
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
annotations:
meta.helm.sh/release-name: stable
meta.helm.sh/release-namespace: default
creationTimestamp: "2021-10-18T10:24:23Z"
generation: 1
labels:
app: kube-prometheus-stack-prometheus
app.kubernetes.io/instance: stable
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/part-of: kube-prometheus-stack
app.kubernetes.io/version: 19.1.0
chart: kube-prometheus-stack-19.1.0
heritage: Helm
release: stable
name: stable-kube-prometheus-sta-prometheus
namespace: default
resourceVersion: "73416"
uid: ec1765a2-cb48-4cd7-8a41-3041a9260e09
spec:
endpoints:
- path: /metrics
port: web
namespaceSelector:
matchNames:
- default
selector:
matchLabels:
app: kube-prometheus-stack-prometheus
release: stable
self-monitor: "true"
Answer the question
In order to leave comments, you need to log in
Prometheus-operator monitors the creation of service monitors in specific namespaces, and filters service monitors by label.
Accordingly, you need to look in the prometheus manifest, what values are set there, and create your own service monitor with the appropriate namespace with the necessary labels.
Thanks to a hint Ivan Koryukov Ivan Koryukov
It seems that I figured it out and everything started up for me.
If anyone is interested here are my
deployment manifests
apiVersion: apps/v1
kind: Deployment
metadata:
name: http-http-deployment
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: /posts/metrics
prometheus.io/port: "8085"
labels:
app: http-http
spec:
replicas: 1
selector:
matchLabels:
app: http-http
template:
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: /posts/metrics
prometheus.io/port: "8085"
labels:
app: http-http
release: stable
spec:
containers:
- name: http-http
image: stasbigunenko/http_http
env:
- name: PORT_HTTP
value: ":8085"
- name: GRPC
value: "grpc-service:9000"
ports:
- containerPort: 8085
apiVersion: v1
kind: Service
metadata:
name: http-monitor-svc
labels:
app: http-http
release: stable
annotations:
prometheus.io/scrape: "true"
prometheus.io/scheme: http
prometheus.io/path: /posts/metrics
prometheus.io/port: "8085"
namespace: default
spec:
ports:
- name: http-http
port: 8085
protocol: TCP
selector:
app: http-http
release: stable
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: http-monitor
labels:
app: http-http
release: stable
namespace: default
spec:
jobLabel: myCounter
endpoints:
- port: http-http
path: posts/metrics
selector:
matchLabels:
app: http-http
release: stable
namespaceSelector:
matchNames:
- default
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question