S
S
SSSSTTTTAAAASSSS2021-10-19 12:30:43
Kubernetes
SSSSTTTTAAAASSSS, 2021-10-19 12:30:43

Kubernetes: unable to add custom service-monitor?

Good afternoon.
I currently have prometheus installed using Helm3 from here.

It seems to work, taking metrics from my pods, which are all in the same node.
The question is that my service at :8085/posts/metrics is already taking metrics in the right format for Prometheus. An exporter is not needed. Accordingly, as far as I understand, I need to add a new service-monitor with the specified metrics reading address.
Unfortunately, I can't find any information on how to do it correctly.
From what I found is this post on stackoverflow:

As far as I understand, I need to add a service first, and then write a separate yaml file for service-monitor so that prometheus and grafana can see them, or can I add information to an existing service that takes metrics, and add service-monitor separately?
I tried to do it by analogy, but somewhere I write something wrong, because does not see prometheus

# Service targeting gitlab instances
apiVersion: v1
kind: Service
metadata:
  name: http-metrics
  labels:
    app: http-metrics
spec:
  ports:
  - name: metrics 
    port: 8085 
    endpoints: /posts
    targetPort: metrics
    protocol: TCP
  selector:
    app: http-metrics
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: http-metrics-servicemonitor
  # Change this to the namespace the Prometheus instance is running in
  # namespace: default
  labels:
    app: http-metrics
    release: stable
spec:
  selector:
    matchLabels:
      app: http-metrics
  endpoints:
  - port: metrics
    interval: 15s


Just in case, the yaml file servicemonitor/stable-kube-prometheus-sta-prometheus
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: stable
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2021-10-18T10:24:23Z"
  generation: 1
  labels:
    app: kube-prometheus-stack-prometheus
    app.kubernetes.io/instance: stable
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: kube-prometheus-stack
    app.kubernetes.io/version: 19.1.0
    chart: kube-prometheus-stack-19.1.0
    heritage: Helm
    release: stable
  name: stable-kube-prometheus-sta-prometheus
  namespace: default
  resourceVersion: "73416"
  uid: ec1765a2-cb48-4cd7-8a41-3041a9260e09
spec:
  endpoints:
  - path: /metrics
    port: web
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels:
      app: kube-prometheus-stack-prometheus
      release: stable
      self-monitor: "true"

Thanks in advance for any help

Answer the question

In order to leave comments, you need to log in

2 answer(s)
I
Ivan Koryukov, 2021-10-19
@SSSSTTTTAAAASSSS

Prometheus-operator monitors the creation of service monitors in specific namespaces, and filters service monitors by label.
Accordingly, you need to look in the prometheus manifest, what values ​​are set there, and create your own service monitor with the appropriate namespace with the necessary labels.

S
SSSSTTTTAAAASSSS, 2021-10-20
@SSSSTTTTAAAASSSS

Thanks to a hint Ivan Koryukov Ivan Koryukov
It seems that I figured it out and everything started up for me.
If anyone is interested here are my
deployment manifests

apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-http-deployment
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: /posts/metrics
    prometheus.io/port: "8085"
  labels:
    app: http-http
spec:
  replicas: 1
  selector:
    matchLabels:
      app: http-http
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: /posts/metrics
        prometheus.io/port: "8085"
      labels:
        app: http-http
        release: stable
    spec:
      containers:
        - name: http-http
          image: stasbigunenko/http_http
          env:
            - name: PORT_HTTP
              value: ":8085"
            - name: GRPC
              value: "grpc-service:9000"
          ports:
            - containerPort: 8085

service
apiVersion: v1
kind: Service
metadata:
  name: http-monitor-svc
  labels:
    app: http-http
    release: stable
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/scheme: http
    prometheus.io/path: /posts/metrics
    prometheus.io/port: "8085"
  namespace: default
spec:
  ports:
    - name: http-http
      port: 8085
      protocol: TCP
  selector:
    app: http-http
    release: stable

servicemonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: http-monitor
  labels:
    app: http-http
    release: stable
  namespace: default
spec:
  jobLabel: myCounter
  endpoints:
    - port: http-http
      path: posts/metrics
  selector:
    matchLabels:
      app: http-http
      release: stable
  namespaceSelector:
    matchNames:
      - default

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question