How to collect metrics from a third party Prometheus server?

Question

Monitoring

Good afternoon!
I ask to prompt to me in what direction to search for the decision.
There is an Openshift cluster, monitoring of which is presented out of the box, as far as I understand (Prometheus).
The cluster is located in a circuit in which I cannot influence the Prom config and generally configure anything (but I have access to their Prometheus web-face).
It was decided to use the federation functionality, which I actually did.
As a target on our Prom server, the address of the "cluster prom" appeared.
But if you look at the metrics that our Prom collects, we will see only prometheus_sd_kubernetes_events_total

Ok, I proceeded from the logic that if I add a third-party Prometheus server to the federation, then I can accordingly get all the metrics from it.

I looked at what is written in the targets on the cluster (third-party) Prometheus:
alertmanager-main (3/3 up)
apiserver (3/3 up)
cluster-monitoring-operator (1/1 up)
kube-controllers (3/3 up)
kube- state-metrics (2/2 up) kubelet
(18/18 up)
node-exporter (9/9 up)
prometheus-k8s (2/2 up)
prometheus-operator (1/1 up)

To collect all the metrics that third-party Prometheus collects, do I need to add each of its targets to my targets?
Then what is the point of federation?

I would be grateful for advice.

Answer 1

The goal of federation is to create a single repository of metrics.
To make it work, as you expect and need to be added to the central Prometheus config, for example:

- job_name: federation
    honor_labels: true
    metrics_path: /federate
    params:
      match[]:
        - '{__name__=~"^[^go_].+"}'
    static_configs:
      - targets:
        - remote-prometheus:9090

__name__ is a hidden label that holds the name of the metric, so all metrics will be collected except for the go_ metrics.
In match, you can also write - '{job=~".+"}' then all metrics from all jobs and other expressions will be collected.
On the target Prometheus, you can add recording rules that will create aggregated metrics.