How to pass text with variables from alertmanager via web-hook?

Question

Monitoring

From Prometheus, an alert of the form flies:

- name: alert.rules
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      description: 'Недоступен сервер {{ $labels.instance }} или сервис {{ $labels.job }} более чем 1 минуту'
      summary: Instance {{ $labels.instance }} down

I made a test webhook in Alertmanager like this:

receivers:
- name: 'webhook'
  webhook_configs:
    - url: 'https://api.telegram.org/bot[ТОКЕН_БОТА]/[email protected][USERNAME_КАНАЛА]&text=тест'

I made a channel in Telegram. The text is sent to it via the API.

Everything is OK, the rule is working, the test message is sent. Now the question itself:
And without bothering with the templates of sent messages, can I just somehow specify the variables received from Prometheus in the Alertmanager config file itself? Instead of "test" I would like to substitute

- url: 'https://api.telegram.org/bot[ТОКЕН_БОТА]/[email protected][USERNAME_КАНАЛА]&text=Упал сервер {{ $labels.instance }}'

Who will prompt how to thrust variables in the given line?

Answer 1

Long communication with colleagues at work and questions in the devops gelochannel helped resolve the issue.
The funny thing is that for almost everyone, the webhook is sent to the telegrambot intermediary program, which already sends the SAME webhook to the telegram API. To the question, why not immediately directly send to telegrams? They answer one thing - they didn’t manage to win initially, they decided to beat them up. Extra traffic, extra point of failure, extra risk somewhere along the way to lose something. Not our way. I show you how to do it right.
1. Create a rule in prometheus. The rule is simple, just for example

# cat /etc/prometheus/rules/alert.rules.yml 
groups:
- name: alert.rules
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 0m
    labels:
      severity: critical
    annotations:
      description: "Server down {{ $labels.instance }} or service {{ $labels.job }}"
      instancename: "{{ $labels.instance }}"
      summary: "Instance down"

We can stuff any tags into labels, which are then available in alertmanager alerts. But only static. Throw a type variable
instancename: {{ $labels.instance }}
I didn't succeed, Prometheus crashes all the time and doesn't start.
2. Then in the alertmanager we set up the native send method. It is also possible through a web hook with a GET request, but there is a limit on the length of the string. You have to be very concise. Here is the simplest verified config:

# cat /etc/alertmanager/alertmanager.yml 
global:

# Отсюда читаем все шаблоны:
  - '/etc/alertmanager/templates/*.tmpl'

route:
  # Группировка алертов
  group_by: ['alertname', 'cluster', 'service']
  # время ожидания перед отправкой уведомления для группы
  group_wait: 10s
  # время отправки повторного сообщения для группы
  group_interval: 10s
  # время до отправки повторного сообщения
  repeat_interval: 10s
  receiver: 'telega'
receivers:
  - name: 'telega'
    telegram_configs:
    - bot_token: '<Здесь указываем ваш bot_token>'
      api_url: 'https://api.telegram.org'
      chat_id: <Здесь указываем ваш chat_id <b>БЕЗ КАВЫЧЕК!!!</b>>
      message:  "Alertname: {{ .GroupLabels.alertname }}\n Severity: {{ .CommonLabels.severity }}\n {{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}"
    <b>  parse_mode: ''</b>

The code sends notifications like this:

monitoring_bot, [02.04.2022 14:38]
Alertname: InstanceDown
 Severity: critical
 Server down 192.168.0.210:8300 or service Consul SD 192.168.0.211

Important!!!
1. specify chat_id without quotes
2. My sending parameters are test, to speed up debugging. You definitely don’t want to see such parameters in real monitoring, change them right away.
3. By default, the telegram_configs method: parses the parameters from prometheus and sends messages only if there was bare text. As soon as variables come across, there will be no messages. Therefore, parsing is required. If the parsing method is empty, then the transmitted data is not parsed and everything is sent.
4. In prometheus, in the annotation section, you can spawn any of your labels like these:

annotations:
      description: "Server down {{ $labels.instance }} or service {{ $labels.job }}"
      instancename: "{{ $labels.instance }}"
      summary: "Instance down"
      aaaaaaaaa: 'XXXXXXXX'
      bbbbbbbb: 'XXXXXXXX'
      cccccccccc: 'XXXXXXXX'

And in alertmanager, access them like this:
{{ range .Alerts }}{{ .Annotations.aaaaaaaaa }}

Answer 2

You can use prometheus_bot there are templates.