Is it possible to send data via node_exporter to the specified prometheus server?

Question

Kubernetes

Simplified scheme for collecting metrics:
(server1:9100) node_exporter<-(server2)prometheus <- (server2)grafana
I need:
(server1) node_exporter -> ??some storage?? (9100)<-(server2)prometheus <- (server2)grafana
Which bundle is used when deploying kubernetes to monitor the state (system, service) in each container or is it not relevant?
Thanks

Answer 1

gremlintv2 ,
in general, purely ideologically, prometheus works through data pull,
but it happens that the entities that we are going to monitor:
1) are ephemeral, that is, they are mortal entities that can rise later, in a different place and with a different address, for example
2) can be just jobs that naturally live for a certain period of time
3) are simply not available for the prometheus server There
are actually 2 good options, as for me
1) if the prometheus sits inside the same cluster, then we simply add a section like this to the config

job_name: ${appName}
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: keep
  regex: ${appName}
  source_labels:
  - __meta_kubernetes_pod_container_name
metrics_path: ${path}
scrape_interval: ${interval}s

then the prometheus will be able to get information about the pods through the native kubernet DNS, filter by __meta_kubernetes_pod_container_name all the necessary ones and everything will be fucked up
2) this option is suitable for both options when the prometheus sits inside or outside the cluster
https://github.com/prometheus/pushgateway
in short 1) you create a pushgateway endpoint for your service, 2) push your metrics there via http 3) in the promiseus-server config you add a `scrape_configs` section as usual, which pulls data from the pushgateway endpoint
PS
node_exporter in this case does not really help because essentially creates an endpoint that should be monitored by the promiteus, if the application itself is not available for the promiteus, then the node_exporter will also not be available, which is obvious