Prometheus 全方位监控实战（四）Pushgateway与服务发现

m0_63859848

391人浏览 · 2026-05-21 20:41:08

m0_63859848 · 2026-05-21 20:41:08 发布

内容概览

模块	核心内容
01 Pushgateway	自定义监控、API管理、TCP状态监控、丢包率监控
02 自定义Exporter	Python开发exporter、Flask集成、Prometheus采集
03 服务发现	基于文件、基于Consul的服务发现
04 联邦模式	分布式采集、数据汇总
05 故障排查	Grafana无数据排查技巧

一、Pushgateway自定义监控

1.1 Pushgateway概述

Prometheus Pushgateway的存在是为了让临时任务和批处理任务能够向Prometheus Server暴露其指标。

由于这类任务可能存在时间不够长，无法被定期抓取，因此它们可以将指标推送到Pushgateway，然后Pushgateway会将这些指标暴露给Prometheus Server。

说白了：Prometheus官方用来短期监控自定义指标的工具。

注意：如果长期监控的组件，建议运维开发人员编写相应的exporters。

GitHub地址：https://github.com/prometheus/pushgateway

1.2 部署Pushgateway组件

# 1. 下载pushgateway
wget https://github.com/prometheus/pushgateway/releases/download/v1.11.2/pushgateway-1.11.2.linux-amd64.tar.gz

# 2. 解压软件包
[root@node-exporter42 ~]# tar xf pushgateway-1.11.2.linux-amd64.tar.gz -C /usr/local/bin/ pushgateway-1.11.2.linux-amd64/pushgateway --strip-components=1
[root@node-exporter42 ~]# ll /usr/local/bin/pushgateway
-rwxr-xr-x 1 1001 1002 21170789 Oct 30 19:57 /usr/local/bin/pushgateway*

# 3. 编写pushgateway的启动脚本
[root@node-exporter42 ~]# cat > /lib/systemd/system/pushgateway.service <<EOF
[Unit]
Description=JasonYin's pushgateway server
After=network.target

[Service]
ExecStart=/bin/bash -c "/usr/local/bin/pushgateway --web.telemetry-path='/metrics' --web.listen-address=:9091 --web.enable-admin-api --persistence.file=/oldboyedu/data/pushgateway.data"
Restart=on-failure
ExecReload=/bin/kill -HUP $MAINPID
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target
EOF

# 4. 启动pushgateway
[root@node-exporter42 ~]# systemctl daemon-reload
[root@node-exporter42 ~]# systemctl enable --now pushgateway.service
[root@node-exporter42 ~]# ss -ntl | grep 9091
LISTEN 0      4096               *:9091            *:*

# 5. 访问pushgateway的WebUI
http://10.0.0.42:9091/#

1.3 Pushgateway的API管理方式

推送数据到Pushgateway

方式一：使用curl推送单值

[root@node-exporter43 ~]# echo "student_online 35" | curl --data-binary @- http://10.0.0.42:9091/metrics/job/oldboyedu_student/instance/10.0.0.43

方式二：使用cat推送多值

[root@node-exporter41 ~]# cat <<EOF | curl --data-binary @- http://10.0.0.42:9091/metrics/job/oldboyedu_disk/instance/10.0.0.41
# HELP linux_cloud_native Number of people learning cloud native in the 103 sessions of Linux.
# TYPE linux_cloud_native counter
linux_cloud_native{auther="Jason Yin","project"="k8s"} 59
linux_cloud_native{auther="Jason Yin","project"="Prometheus"} 56
linux_cloud_native{auther="Jason Yin","project"="ceph"} 52
linux_cloud_native{auther="Jason Yin","project"="elasticstack"} 55
linux_cloud_native{auther="Jason Yin","project"="docker"} 80
# HELP disk_usage Disk usage.
# TYPE disk_usage gauge
disk_usage 92.56
EOF

方式三：使用echo发送多个数据

[root@node-exporter41 ~]# echo """
# TYPE student_online gauge
# HELP student_online Number of students online.
student_online 150""" | curl --data-binary @- http://10.0.0.42:9091/metrics/job/oldboyedu_student/instance/10.0.0.41

获取监控数据

[root@node-exporter41 ~]# curl -s http://10.0.0.42:9091/metrics | egrep "cloud|disk_usage|student_online"
# HELP disk_usage Disk usage.
# TYPE disk_usage gauge
disk_usage{instance="10.0.0.41",job="oldboyedu_disk"} 92.56
# HELP linux_cloud_native Number of people learning cloud native in the 102 sessions of Linux.
# TYPE linux_cloud_native counter
linux_cloud_native{auther="Jason Yin",instance="10.0.0.41",job="oldboyedu_disk",project="Prometheus"} 56
...

删除监控数据

# 删除指定的job，删不掉
[root@node-exporter41 ~]# curl -X DELETE http://10.0.0.42:9091/metrics/job/oldboyedu_disk

# 删除时必须指定的job和instance
[root@node-exporter41 ~]# curl -X DELETE http://10.0.0.42:9091/metrics/job/oldboyedu_disk/instance/10.0.0.41

# 删除pushgateway的所有指标
[root@node-exporter41 ~]# curl -X PUT http://10.0.0.42:9091/api/v1/admin/wipe

温馨提示：如果要删除所有指标，也可以直接重启pushgateway，因为重启后数据丢失。

1.4 Pushgateway监控在线课程人数案例

# 1. 准备测试脚本
[root@node-exporter41 ~]# cat /tmp/push-student-online.sh
#!/bin/bash

NUMBER_OF_STUDENT=("$((RANDOM%51+50))" "$((RANDOM%51+50))" "$((RANDOM%51+50))" "$((RANDOM%51+50))" "$((RANDOM%51+50))")

cat <<EOF | curl --data-binary @- http://10.0.0.42:9091/metrics/job/student_online/instance/10.0.0.41
# HELP linux_cloud_native Number of people learning cloud native in the 102 sessions of Linux.
# TYPE linux_cloud_native counter
linux_cloud_native{auther="Jason Yin","project"="k8s"} ${NUMBER_OF_STUDENT[0]}
linux_cloud_native{auther="Jason Yin","project"="Prometheus"} ${NUMBER_OF_STUDENT[1]}
linux_cloud_native{auther="Jason Yin","project"="ceph"} ${NUMBER_OF_STUDENT[2]}
linux_cloud_native{auther="Jason Yin","project"="elasticstack"} ${NUMBER_OF_STUDENT[3]}
linux_cloud_native{auther="Jason Yin","project"="docker"} ${NUMBER_OF_STUDENT[4]}
EOF

[root@node-exporter41 ~]# chmod +x /tmp/push-student-online.sh

# 2. 添加定时任务（每分钟执行）
[root@node-exporter41 ~]# echo "* * * * * /tmp/push-student-online.sh" > /var/spool/cron/crontabs/root
[root@node-exporter41 ~]# crontab -l
* * * * * /tmp/push-student-online.sh

# 如果需要精确到秒执行，可以写死循环
[root@node-exporter41 ~]# cat /tmp/patch-push.sh
#!/bin/bash
while true 
  do
   sleep 3
   /tmp/push-student-online.sh
done

[root@node-exporter41 ~]# bash /tmp/patch-push.sh

Prometheus配置

# prometheus.yml
- job_name: "oldboyedu-pushgateway"
  honor_labels: true  # 采集数据标签冲突时，远程标签覆盖本地标签
  static_configs:
    - targets:
        - 10.0.0.42:9091

# 热加载配置
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload

Grafana配置

配置项	值
PromQL	`linux_cloud_native{project="$subject"}`
变量subject	Custom: k8s,Prometheus,elasticstack,ceph
Legend	Custom: `$subject课程在线人数`

新建dashboard，添加课程变量

设置

1.5 Pushgateway监控TCP的12种状态

# 1. 创建TCP状态监控脚本
[root@prometheus-server31 ~]# cat > /usr/local/bin/tcp_status.sh <<'EOF'
#!/bin/bash

pushgateway_url="http://10.0.0.42:9091/metrics/job/tcp_status/instance/10.0.0.31"

state="SYN-SENT SYN-RECV FIN-WAIT-1 FIN-WAIT-2 TIME-WAIT CLOSE CLOSE-WAIT LAST-ACK LISTEN CLOSING ESTAB UNKNOWN"

echo """
# TYPE tcp_connections gauge
# HELP tcp_connections Number of TCP state connections.""" > /tmp/tcp.txt

for i in $state
  do
    count=`ss -tan | grep $i | wc -l`
    echo tcp_connections{state=\""$i"\"} $count >> /tmp/tcp.txt
done;

cat /tmp/tcp.txt | curl --data-binary @- $pushgateway_url
EOF

# 2. 调用脚本
[root@prometheus-server31 ~]# bash /usr/local/bin/tcp_status.sh

# 3. Prometheus查询数据
tcp_connections

Grafana配置同上新建变量，设置

配置项	值
PromQL	`tcp_connections{state="$state"}`
变量state	Custom: SYN-SENT,SYN-RECV,FIN-WAIT-1,FIN-WAIT-2,TIME-WAIT,CLOSE,CLOSE-WAIT,LAST-ACK,LISTEN,CLOSING,ESTAB,UNKNOWN
Legend	Custom: `{{state}}连接数`

1.6 完成网络的丢包率监控

# 1. 创建丢包率监控脚本
[root@prometheus-server31 ~]# cat > /usr/local/bin/loss_packet.sh <<'EOF'
#!/bin/bash

host="www.oldboyedu.com"

loss=`ping $host -c 10 | grep packet | awk '{print $6}' | tr -d '%'`

echo """
# TYPE loss_packet gauge
# HELP loss_packet ${host} Packet Loss Rate.
loss_packet $loss""" | curl --data-binary @- http://10.0.0.42:9091/metrics/job/oldboyedu_loss_packet/instance/${host}
EOF

[root@prometheus-server31 ~]# bash /usr/local/bin/loss_packet.sh

Grafana配置

配置项	值
变量target	Query → Label values → instance → {job=“oldboyedu_loss_packet”}
PromQL	`loss_packet{instance="$target"}`
Legend	Custom: `$target丢包率`

二、开发Exporter实现自定义指标监控

2.1 为什么要自定义Exporter

Pushgateway存在的缺陷是重启后数据丢失，需要用户编写脚本推送到Pushgateway，使用起来相对麻烦。

生产环境中，我们可以考虑让运维开发人员编写Python、Golang程序实现自定义Exporter来暴露指标。

2.2 使用Python程序自定义Exporter案例

# 1. 安装pip3工具包
[root@node-exporter43 ~]# apt update
[root@node-exporter43 ~]# apt install -y python3-pip

# 2. 安装相关模块库
[root@node-exporter43 ~]# pip3 install flask prometheus_client -i https://mirrors.aliyun.com/pypi/simple

# 3. 编写代码
# Python Flask Web应用 + Prometheus监控的混合程序
# 同时启动了两个服务：
#		Web API服务 (端口8001) - 提供业务接口
#		Prometheus监控服务 (端口8000) - 暴露监控指标
[root@node-exporter43 ~]# cat > flask_metric.py <<'EOF'
#!/usr/bin/python3
# auther: Jason Yin
# blog: https://www.cnblogs.com/yinzhengjie/

from prometheus_client import start_http_server, Counter, Summary
from flask import Flask, jsonify
from wsgiref.simple_server import make_server
import time

app = Flask(__name__)

# Create a metric to track time spent and requests made
REQUEST_TIME = Summary('request_processing_seconds', 'Time spent processing request')
COUNTER_TIME = Counter("request_count", "Total request count of the host")

@app.route("/apps")
@REQUEST_TIME.time()
def requests_count():
    COUNTER_TIME.inc()
    return jsonify({"office": "https://www.oldboyedu.com"}, {"auther": "Jason Yin"})

if __name__ == "__main__":
    print("启动教育程序: oldboyedu-linux-python-exporter, 访问路径: http://0.0.0.0:8001/apps，监控服务: http://0.0.0.0:8000")
    start_http_server(8000)
    httpd = make_server('0.0.0.0', 8001, app)
    httpd.serve_forever()
EOF

# 4. 启动python程序
[root@node-exporter43 ~]# python3 flask_metric.py
启动教育程序: oldboyedu-linux-python-exporter, 访问路径: http://0.0.0.0:8001/apps，监控服务: http://0.0.0.0:8000

# 5. 客户端测试脚本
[root@node-exporter42 ~]# cat > oldboyedu_curl_metrics.sh <<'EOF'
#!/bin/bash

URL=http://10.0.0.43:8001/apps

while true;do
    curl_num=$(( $RANDOM%50+1 ))
    sleep_num=$(( $RANDOM%5+1 ))
    for c_num in `seq $curl_num`;do
        curl -s $URL &> /dev/null
    done
    sleep $sleep_num
done
EOF

[root@node-exporter42 ~]# bash oldboyedu_curl_metrics.sh

# 6. 访问WebUI
http://10.0.0.43:8001/apps   # 实际业务，对外提供的服务
http://10.0.0.43:8000/metrics  # 实际业务的指标监控

2.3 Prometheus监控Python自定义Exporter

# 1. 编辑配置文件
[root@prometheus-server31 ~]# vim /oldboyedu/softwares/prometheus-2.53.4.linux-amd64/prometheus.yml

- job_name: "python_custom_metrics"
  static_configs:
    - targets:
      - 10.0.0.43:8000

# 2. 检查配置文件语法
[root@prometheus-server31 prometheus-2.53.4.linux-amd64]# ./promtool check config prometheus.yml
Checking prometheus.yml
 SUCCESS: prometheus.yml is valid prometheus config file syntax

# 3. 重新加载配置文件
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload

# 4. 验证prometheus是否采集到数据
http://10.0.0.31:9090/targets?search=

Grafana 中使用的 PromQL语句

指标	PromQL	说明
请求总数	`request_count_total`	apps请求总数
QPS	`increase(request_count_total{job="python_custom_metrics"}[1m])`	每分钟请求数量曲线
变化率	`irate(request_count_total{job="python_custom_metrics"}[1m])`	每分钟请求量变化率曲线
平均耗时	`request_processing_seconds_sum / request_processing_seconds_count`	每分钟请求处理平均耗时

三、Prometheus服务发现

3.1 服务发现概述

基于文件和consul的服务发现的区别

维度	基于文件的服务发现 (File SD)	基于 Consul 的服务发现 (Consul SD)
工作方式	文件接口。你需要通过脚本、配置管理工具等，将要监控的目标写入一个特定格式的文件。	注册中心。各监控目标（服务）主动调用 Consul API 进行“注册”或“注销”。
依赖组件	无。仅依赖文件系统，不需要额外组件。	有。需要独立部署和维护一套 Consul 集群。
实时性	较高。Prometheus 通过 `<font style="color:rgb(15, 17, 21);background-color:rgb(235, 238, 242);">inotify</font>` 机制监控文件变化，变更后可立刻生效。	高。服务状态变化会实时反馈在 Consul 中，Prometheus 轮询 Consul 获取最新状态。
操作方式	运维在服务端操作。需要登录到 Prometheus 服务器，修改目标文件。	自动化操作。通过调用 Consul API 即可完成，非常适合集成到 CI/CD 或自动化部署流程中。
配置格式	支持 YAML 或 JSON 格式。	通过 Consul API 进行配置。
适用场景	中小型环境，或已经使用 Ansible、SaltStack 等自动化工具管理配置的场景。	大型、动态的微服务架构，追求高度自动化、服务生命周期频繁变化的场景。

Prometheus支持动态配置和静态配置：

配置类型	说明	特点
静态配置（static_configs）	修改配置文件后需要热加载或重启	配置简单
动态配置（*_sd_config）	无需重启，监听文件或注册中心	自动化程度高

常见的服务发现配置：

file_sd_config：基于文件的服务发现
consul_sd_config：基于Consul的服务发现
kubernetes_sd_config：基于K8s的服务发现

参考链接：

https://prometheus.io/docs/prometheus/2.55/configuration/configuration/

3.2 基于文件的服务发现案例

# 1. 修改配置文件
[root@prometheus-server31 ~]# vim /oldboyedu/softwares/prometheus-2.53.4.linux-amd64/prometheus.yml

- job_name: "oldboyedu-file-sd"
  file_sd_configs:
    - files:
        - /tmp/xixi.json
        - /tmp/haha.yaml

# 2. 热加载配置文件
[root@prometheus-server31 prometheus-2.53.4.linux-amd64]# curl -X POST http://10.0.0.31:9090/-/reload

# 3. 编写json格式文件
[root@prometheus-server31 ~]# cat > /tmp/xixi.json <<EOF
[
  {
    "targets": [ "10.0.0.41:9100" ],
    "labels": {
      "school": "oldboyedu", 
      "class": "linux103"
    }
  }
]
EOF

# 4. 验证是否自动监控目标
http://10.0.0.31:9090/targets?search=

# 5. 编写yaml文件
[root@prometheus-server31 ~]# cat > /tmp/haha.yaml <<EOF
- targets:
  - '10.0.0.42:9100' 
  - '10.0.0.43:9100' 
  labels:
    address: ShaHe
    ClassRoom: JiaoShi02
EOF

# 6. 验证是否自动监控目标
http://10.0.0.31:9090/targets?search=

3.3 基于Consul的服务发现案例

部署Consul集群

官方文档：https://www.consul.io/

# 1. 下载consul软件包
wget https://releases.hashicorp.com/consul/1.22.7/consul_1.22.7_linux_amd64.zip

# 2. 解压consul
[root@node-exporter41 ~]# unzip consul_1.22.7_linux_amd64.zip -d /usr/local/bin/

# 分发到其他节点
[root@node-exporter41 ~]# scp /usr/local/bin/consul 10.0.0.42:/usr/local/bin/
[root@node-exporter41 ~]# scp /usr/local/bin/consul 10.0.0.43:/usr/local/bin/

# 3. 运行consul集群
# 服务端43:
[root@node-exporter43 ~]# consul agent -server -bootstrap -bind=10.0.0.43 -data-dir=/oldboyedu/softwares/consul -client=10.0.0.43 -ui

# 客户端42:
[root@node-exporter42 ~]# consul agent -server -bind=10.0.0.42 -data-dir=/oldboyedu/softwares/consul -client=10.0.0.42 -ui -retry-join=10.0.0.43

# 客户端41:
[root@node-exporter41 ~]# consul agent -server -bind=10.0.0.41 -data-dir=/oldboyedu/softwares/consul -client=10.0.0.41 -ui -retry-join=10.0.0.43

# 4. 查看各节点的监听端口
ss -ntl | egrep "8300|8500"

# 5. 访问consul服务的WebUI
http://10.0.0.43:8500/ui/dc1/nodes

基于Consul的服务发现案例

# 1. 修改prometheus的配置文件
[root@prometheus-server31 ~]# vim /oldboyedu/softwares/prometheus-2.53.4.linux-amd64/prometheus.yml

- job_name: "oldboyedu-consul-service-discovery"
  # 配置基于consul的服务发现
  consul_sd_configs:
    - server: 10.0.0.43:8500
    - server: 10.0.0.42:8500
    - server: 10.0.0.41:8500
  relabel_configs:
    # 匹配consul的源标签字段，表示服务名称
    - source_labels: [__meta_consul_service]
      regex: consul
      action: drop

# 2. 检查配置文件是否正确
[root@prometheus-server31 ~]# cd /oldboyedu/softwares/prometheus-2.53.4.linux-amd64/
[root@prometheus-server31 prometheus-2.53.4.linux-amd64]# ./promtool check config ./prometheus.yml
Checking ./prometheus.yml
 SUCCESS: ./prometheus.yml is valid prometheus config file syntax

# 3. 重新加载配置
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload

# 4. 被监控节点注册到consul集群
[root@prometheus-server31 ~]# curl -X PUT -d '{"id":"prometheus-node42","name":"oldboyedu-prometheus-node42","address":"10.0.0.42","port":9100,"tags":["node-exporter"],"checks": [{"http":"http://10.0.0.42:9100","interval":"5m"}]}' http://10.0.0.43:8500/v1/agent/service/register

curl -X PUT -d '{"id":"prometheus-node43","name":"oldboyedu-prometheus-node43","address":"10.0.0.43","port":9100,"tags":["node-exporter"],"checks": [{"http":"http://10.0.0.43:9100","interval":"5m"}]}' http://10.0.0.43:8500/v1/agent/service/register

# 5. 注销节点
[root@prometheus-server31 ~]# curl -X PUT http://10.0.0.43:8500/v1/agent/service/deregister/prometheus-node42

# 6. 验证配置是否生效
http://10.0.0.31:9090/targets?search=

3.4 Prometheus监控Consul应用

# 1. 下载consul_exporter
wget https://github.com/prometheus/consul_exporter/releases/download/v0.13.0/consul_exporter-0.13.0.linux-amd64.tar.gz

# 2. 解压软件包
[root@node-exporter42 ~]# tar xf consul_exporter-0.13.0.linux-amd64.tar.gz -C /usr/local/bin/ consul_exporter-0.13.0.linux-amd64/consul_exporter --strip-components=1
[root@node-exporter42 ~]# ll /usr/local/bin/consul_exporter
-rwxr-xr-x 1 1001 1002 19294344 Nov  6  2024 /usr/local/bin/consul_exporter*

# 3. 运行consul_exporter
[root@node-exporter42 ~]# consul_exporter --consul.server="http://10.0.0.41:8500" --web.telemetry-path="/metrics" --web.listen-address=:9107

# 4. 访问consul_exporter的webUI
http://10.0.0.42:9107/metrics

# 5. prometheus server监控consul_exporter
[root@prometheus-server31 ~]# curl -X PUT -d '{"id":"consul-server","name":"node-exporter42","address":"10.0.0.42","port":9107,"tags":["consul-cluster"],"checks": [{"http":"http://10.0.0.42:9107","interval":"5m"}]}' http://10.0.0.43:8500/v1/agent/service/register

# 6. 验证配置是否生效
http://10.0.0.31:9090/targets

# 7. Grafana导入模板ID
12049

四、Prometheus联邦模式

4.1 为什么需要联邦模式

联邦模式主要作用是为了减轻Prometheus Server的 I/O 压力，实现分布式存储数据。
如上图所示，设计3台Prometheus Server，其中2台负责采集数据，1台负责数据汇总。
这里新加入两台虚拟机10.0.0.32，10.0.0.33，分别安装好Prometheus。
10.0.0.32 以文件发现的方式监控41。
10.0.0.33 以 consol 发现的方式监控42、43。

4.2 Prometheus 32节点监控41节点

# 1. 安装软件包
wget https://github.com/prometheus/prometheus/releases/download/v3.5.3/prometheus-3.5.3.linux-amd64.tar.gz

tar xf prometheus-3.5.3.linux-amd64.tar.gz -C /usr/local/

cd /usr/local/prometheus-3.5.3.linux-amd64/
./prometheus

# 2. 访问WebUI
http://10.0.0.31:9090/

# 3. 修改prometheus的配置文件
[root@prometheus-server32 ~]# cd /oldboyedu/softwares/prometheus-2.53.4.linux-amd64/
[root@prometheus-server32 prometheus-2.53.4.linux-amd64]# tail -5 prometheus.yml

- job_name: "oldboyedu-file-sd-yaml-node_exporter"
  file_sd_configs:
    - files:
        - /oldboyedu/softwares/prometheus-2.53.4.linux-amd64/sd/*.yaml

# 4. 创建服务发现的文件
[root@prometheus-server32 prometheus-2.53.4.linux-amd64]# mkdir sd
[root@prometheus-server32 prometheus-2.53.4.linux-amd64]# cat sd/oldboyedu-linux.yaml
- targets:
    - '10.0.0.41:9100'
  labels:
    school: oldboyedu
    class: linux103

# 5. 检查配置文件
[root@prometheus-server32 prometheus-2.53.4.linux-amd64]# ./promtool check config prometheus.yml
Checking prometheus.yml
 SUCCESS: prometheus.yml is valid prometheus config file syntax

# 6. 热加载配置
[root@prometheus-server32 prometheus-2.53.4.linux-amd64]# curl -X POST 10.0.0.32:9090/-/reload

# 7. 验证服务是否生效
http://10.0.0.32:9090/targets

4.3 Prometheus 33节点监控42、43节点

# 1. 安装软件包
wget https://github.com/prometheus/prometheus/releases/download/v3.5.3/prometheus-3.5.3.linux-amd64.tar.gz

tar xf prometheus-3.5.3.linux-amd64.tar.gz -C /usr/local/

cd /usr/local/prometheus-3.5.3.linux-amd64/
./prometheus

# 2. 访问WebUI
http://10.0.0.31:9090/

# 4. 修改prometheus的配置文件
[root@prometheus-server33 ~]# cd /oldboyedu/softwares/prometheus-2.53.4.linux-amd64/
[root@prometheus-server33 prometheus-2.53.4.linux-amd64]# tail prometheus.yml

- job_name: "oldboyedu-consul-sd-node_exporter"
  consul_sd_configs:
    - server: 10.0.0.43:8500
    - server: 10.0.0.42:8500
    - server: 10.0.0.41:8500
  relabel_configs:
    - source_labels: [__meta_consul_service]
      regex: consul
      action: drop

# 5. 检查配置文件
[root@prometheus-server33 prometheus-2.53.4.linux-amd64]# ./promtool check config prometheus.yml
Checking prometheus.yml
 SUCCESS: prometheus.yml is valid prometheus config file syntax

# 6. 热加载配置
[root@prometheus-server33 prometheus-2.53.4.linux-amd64]# curl -X POST 10.0.0.33:9090/-/reload

# 7. 42和43注册节点
[root@node-exporter41 ~]# curl -X PUT -d '{"id":"prometheus-node42","name":"oldboyedu-prometheus-node42","address":"10.0.0.42","port":9100,"tags":["node-exporter"],"checks": [{"http":"http://10.0.0.42:9100","interval":"5m"}]}' http://10.0.0.43:8500/v1/agent/service/register

curl -X PUT -d '{"id":"prometheus-node43","name":"oldboyedu-prometheus-node43","address":"10.0.0.43","port":9100,"tags":["node-exporter"],"checks": [{"http":"http://10.0.0.43:9100","interval":"5m"}]}' http://10.0.0.43:8500/v1/agent/service/register

# 8. 验证服务是否生效
http://10.0.0.33:9090/targets

4.4 Prometheus 31节点配置联邦模式

# 1. 修改配置文件，注册掉所有job（保留localhost），这里31只监控32和33，汇总数据给gragana
[root@prometheus-server31 ~]# vim /oldboyedu/softwares/prometheus-2.53.4.linux-amd64/prometheus.yml

- job_name: "oldboyedu-prometheus-federate-32"
  metrics_path: "/federate"
  honor_labels: true
  params:
    "match[]":
    - '{job="prometheus"}'
    - '{__name__=~"job:.*"}'
    - '{__name__=~"node.*"}'
  static_configs:
    - targets:
        - "10.0.0.32:9090"

- job_name: "oldboyedu-prometheus-federate-33"
  metrics_path: "/federate"
  honor_labels: true
  params:
    "match[]":
    - '{job="prometheus"}'
    - '{__name__=~"job:.*"}'
    - '{__name__=~"node.*"}'
    - '{__name__=~"consul.*"}'
  static_configs:
    - targets:
        - "10.0.0.33:9090"

# 2. 检查配置文件
[root@prometheus-server31 ~]# cd /oldboyedu/softwares/prometheus-2.53.4.linux-amd64/
[root@prometheus-server31 prometheus-2.53.4.linux-amd64]# ./promtool check config prometheus.yml
Checking prometheus.yml
 SUCCESS: prometheus.yml is valid prometheus config file syntax

# 3. 热加载配置文件
[root@prometheus-server31 prometheus-2.53.4.linux-amd64]# curl -X POST http://10.0.0.31:9090/-/reload

# 4. 验证服务是否生效
http://10.0.0.31:9090/targets?search=

# 基于如下的PromQL查询:
node_cpu_seconds_total{mode="idle",job=~"oldboyedu.*sd.*"}[20s]

# 5. Grafana导入模板ID
1860

五、Grafana出图故障排查技巧

如上图所示，可以根据数据流走向来排查为什么Grafana查询不到数据。

以用户自定义的监控数据推送到Pushgateway为例，排查思路如下：

步骤	检查项	命令/方法
1	数据是否推送到Pushgateway	`curl -s http://pushgateway:9091/metrics
2	Prometheus是否能查询到数据	Prometheus WebUI → Graph → 查询指标
3	Grafana数据源配置	Configuration → Data Sources → Test
4	变量引用问题	检查Dashboard变量是否匹配
5	PromQL语句	Explore页面测试查询
6	时间选择范围	调整Dashboard时间范围