📋 方案概述

基于你的需求，我将提供一个完整的、可分阶段实施的监控方案。你可以先安装部分节点完成功能测试，再逐步扩展到全量节点。

整体架构

🚀 第一阶段：基础环境搭建（1天）

1.1 部署Consul集群（3节点）

在三台机器上执行以下操作（假设IP：192.168.1.10, 192.168.1.11, 192.168.1.12）：

bash

# 下载安装Consul
wget https://releases.hashicorp.com/consul/1.18.0/consul_1.18.0_linux_amd64.zip
unzip consul_1.18.0_linux_amd64.zip
sudo mv consul /usr/local/bin/

# 创建目录和用户
sudo mkdir -p /etc/consul.d /var/lib/consul
sudo useradd --system --home /etc/consul.d --shell /bin/false consul
sudo chown -R consul:consul /etc/consul.d /var/lib/consul

# 创建配置文件（根据实际IP修改）
sudo tee /etc/consul.d/server.hcl <<EOF
datacenter = "dc1"
data_dir = "/var/lib/consul"
server = true
bootstrap_expect = 3
client_addr = "0.0.0.0"
bind_addr = "0.0.0.0"
advertise_addr = "192.168.1.10"  # 改为本机IP
ui_config { enabled = true }
retry_join = ["192.168.1.10", "192.168.1.11", "192.168.1.12"]
ports { http = 8500 grpc = 8502 }
acl = {
  enabled = true
  default_policy = "deny"
  enable_token_persistence = true
}
EOF

# 创建systemd服务
sudo tee /etc/systemd/system/consul.service <<EOF
[Unit]
Description=Consul Agent
After=network.target

[Service]
User=consul
Group=consul
ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d
ExecReload=/bin/kill -HUP \$MAINPID
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

# 启动Consul
sudo systemctl daemon-reload
sudo systemctl enable consul
sudo systemctl start consul

1.2 初始化ACL并获取Token

在任意一台Consul Server上执行：

bash

# 获取bootstrap token
consul acl bootstrap

# 输出示例（保存好SecretID！）：
# SecretID: 87654321-4321-4321-4321-cba987654321
export CONSUL_TOKEN="87654321-4321-4321-4321-cba987654321"

1.3 部署Prometheus

bash

# 下载安装Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz
tar xvf prometheus-2.45.0.linux-amd64.tar.gz
sudo mv prometheus-2.45.0.linux-amd64 /opt/prometheus

# 创建数据目录和用户
sudo mkdir -p /var/lib/prometheus
sudo useradd --system --no-create-home prometheus
sudo chown -R prometheus:prometheus /opt/prometheus /var/lib/prometheus

# 创建基础配置文件（后面会逐步完善）
sudo tee /opt/prometheus/prometheus.yml <<EOF
global:
  scrape_interval: 30s
  evaluation_interval: 30s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
EOF

# 创建systemd服务
sudo tee /etc/systemd/system/prometheus.service <<EOF
[Unit]
Description=Prometheus
After=network.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/opt/prometheus/prometheus \\
  --config.file=/opt/prometheus/prometheus.yml \\
  --storage.tsdb.path=/var/lib/prometheus \\
  --web.listen-address=:9090
Restart=always

[Install]
WantedBy=multi-user.target
EOF

# 启动Prometheus
sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus

1.4 部署Grafana

bash

# 安装Grafana
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install -y grafana

# 启动
sudo systemctl enable grafana-server
sudo systemctl start grafana-server

# 访问：http://你的PrometheusIP:3000
# 默认用户名/密码：admin/admin

1.5 部署Alertmanager

bash

# 下载安装Alertmanager
wget https://github.com/prometheus/alertmanager/releases/download/v0.25.0/alertmanager-0.25.0.linux-amd64.tar.gz
tar xvf alertmanager-0.25.0.linux-amd64.tar.gz
sudo mv alertmanager-0.25.0.linux-amd64 /opt/alertmanager

# 创建配置文件
sudo tee /opt/alertmanager/alertmanager.yml <<EOF
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'env', 'role']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'webhook'

receivers:
  - name: 'webhook'
    webhook_configs:
      - url: 'http://localhost:8080/alert'  # 你的告警接收地址

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'env']
EOF

# 创建systemd服务
sudo tee /etc/systemd/system/alertmanager.service <<EOF
[Unit]
Description=Alertmanager
After=network.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/opt/alertmanager/alertmanager \\
  --config.file=/opt/alertmanager/alertmanager.yml \\
  --storage.path=/var/lib/alertmanager
Restart=always

[Install]
WantedBy=multi-user.target
EOF

# 启动
sudo systemctl daemon-reload
sudo systemctl enable alertmanager
sudo systemctl start alertmanager

1.6 验证基础环境

bash

# 验证Consul
curl http://localhost:8500/v1/status/leader
curl -H "X-Consul-Token: $CONSUL_TOKEN" http://localhost:8500/v1/catalog/services

# 验证Prometheus
curl http://localhost:9090/api/v1/targets

# 验证Grafana
curl http://localhost:3000/api/health

# 验证Alertmanager
curl http://localhost:9093/api/v2/status

🧪 第二阶段：测试节点部署（半天）

2.1 Node Exporter部署方式

Node Exporter有两种启动方式，根据你的环境选择 ：

方式A：systemd服务（推荐，生产环境）

bash

# 下载安装Node Exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xvf node_exporter-1.7.0.linux-amd64.tar.gz
sudo mv node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/

# 创建专用用户
sudo useradd -rs /bin/false node_exporter

# 创建systemd服务
sudo tee /etc/systemd/system/node_exporter.service <<EOF
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \\
  --web.listen-address=:9100 \\
  --path.rootfs=/
Restart=always

[Install]
WantedBy=multi-user.target
EOF

# 启动
sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

# 验证
curl http://localhost:9100/metrics | head -n 5
方式B：Docker方式（测试环境）

bash

# 使用Docker运行
docker run -d \
  --name node_exporter \
  --restart always \
  --net="host" \
  --pid="host" \
  -v "/:/host:ro,rslave" \
  prom/node-exporter:latest \
  --path.rootfs=/host

2.2 关于克隆的说明

Node Exporter支持克隆/镜像部署，没有client ID限制。你可以：

  1. 制作一个包含Node Exporter的黄金镜像

  2. 使用clustershell批量分发

  3. 新克隆的机器会自动注册到Consul

注意事项：

  • 每台机器需要有唯一的主机名（用于生成唯一的Service ID）

  • Consul注册脚本中的service.id必须唯一（我们使用node-exporter-$(hostname)确保唯一性）

2.3 创建Consul注册脚本

bash

# /tmp/register-to-consul.sh
cat > /tmp/register-to-consul.sh <<'EOF'
#!/bin/bash
CONSUL_SERVER="192.168.1.10:8500"  # 你的Consul Server IP
TOKEN="87654321-4321-4321-4321-cba987654321"

# 获取基础信息
PRIVATE_IP=$(hostname -I | awk '{print $1}')
HOSTNAME=$(hostname)

# 构造服务定义
cat > /tmp/service.json <<SERVICE_EOF
{
  "ID": "node-exporter-$HOSTNAME",
  "Name": "node_exporter",
  "Tags": ["huaweicloud", "test"],  # 测试节点打test标签
  "Address": "$PRIVATE_IP",
  "Port": 9100,
  "Meta": {
    "hostname": "$HOSTNAME",
    "env": "test",
    "role": "test-node",
    "registered_at": "$(date +%Y%m%d-%H%M%S)"
  },
  "Check": {
    "http": "http://$PRIVATE_IP:9100/metrics",
    "interval": "30s",
    "timeout": "5s",
    "deregister_critical_service_after": "10m"
  }
}
SERVICE_EOF

# 注册到Consul
curl -X PUT \
  -H "X-Consul-Token: $TOKEN" \
  --data @/tmp/service.json \
  http://$CONSUL_SERVER/v1/agent/service/register

echo "$(date): 注册 $HOSTNAME ($PRIVATE_IP)"
EOF

chmod +x /tmp/register-to-consul.sh

2.4 在3台测试节点上执行

使用clustershell批量操作：

bash

# 分发注册脚本
clush -w 192.168.1.101,192.168.1.102,192.168.1.103 -c /tmp/register-to-consul.sh --dest=/tmp/

# 执行注册
clush -w 192.168.1.101,192.168.1.102,192.168.1.103 "bash /tmp/register-to-consul.sh"

# 添加到crontab（每5分钟重新注册）
clush -w 192.168.1.101,192.168.1.102,192.168.1.103 \
  'echo "*/5 * * * * root bash /tmp/register-to-consul.sh" >> /etc/crontab'

2.5 验证测试节点注册

bash

# 查看所有注册的服务
curl -H "X-Consul-Token: $CONSUL_TOKEN" http://localhost:8500/v1/catalog/services

# 查看node_exporter的测试节点
curl -H "X-Consul-Token: $CONSUL_TOKEN" \
  "http://localhost:8500/v1/health/service/node_exporter?tag=test&pretty"

📊 第三阶段：Prometheus配置与资源池分组

3.1 完整Prometheus配置（带分组）

更新 /opt/prometheus/prometheus.yml：

yaml

global:
  scrape_interval: 30s
  evaluation_interval: 30s
  external_labels:
    monitor: 'huawei-cloud-ecs'

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

rule_files:
  - "alerts/*.yml"

scrape_configs:
  # ============ 1. 监控Prometheus自身 ============
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
        labels:
          app: 'prometheus'

  # ============ 2. 监控Consul集群 ============
  - job_name: 'consul'
    static_configs:
      - targets: ['192.168.1.10:8500', '192.168.1.11:8500', '192.168.1.12:8500']
    metrics_path: /v1/agent/metrics
    params:
      format: ['prometheus']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

  # ============ 3. 动态发现所有ECS节点 ============
  - job_name: 'huawei-ecs-all'
    consul_sd_configs:
      - server: '192.168.1.10:8500'
        services: ['node_exporter']
        token: '87654321-4321-4321-4321-cba987654321'
        refresh_interval: 30s

    relabel_configs:
      # 只保留健康检查通过的实例
      - source_labels: [__meta_consul_health]
        regex: 'passing'
        action: keep

      # 提取IP和端口
      - source_labels: [__meta_consul_address, __meta_consul_service_port]
        separator: ':'
        target_label: '__address__'

      # 提取元数据作为标签
      - source_labels: [__meta_consul_metadata_hostname]
        target_label: 'hostname'
      - source_labels: [__meta_consul_metadata_env]
        target_label: 'env'
      - source_labels: [__meta_consul_metadata_role]
        target_label: 'role'

      # 从Tags中提取标签
      - source_labels: [__meta_consul_tags]
        regex: '.*,prod,.*'
        target_label: 'env_from_tag'
        replacement: 'prod'
      - source_labels: [__meta_consul_tags]
        regex: '.*,staging,.*'
        target_label: 'env_from_tag'
        replacement: 'staging'
      - source_labels: [__meta_consul_tags]
        regex: '.*,test,.*'
        target_label: 'env_from_tag'
        replacement: 'test'

      # 添加静态标签
      - target_label: 'cloud_provider'
        replacement: 'huaweicloud'
      - target_label: 'discovery'
        replacement: 'consul'

  # ============ 4. 按环境分组：生产环境 ============
  - job_name: 'huawei-ecs-prod'
    consul_sd_configs:
      - server: '192.168.1.10:8500'
        services: ['node_exporter']
        token: '87654321-4321-4321-4321-cba987654321'
        tags: ['prod']  # 只发现带prod标签的实例

    relabel_configs:
      - source_labels: [__meta_consul_health]
        regex: 'passing'
        action: keep
      - source_labels: [__meta_consul_address, __meta_consul_service_port]
        separator: ':'
        target_label: '__address__'
      - source_labels: [__meta_consul_metadata_role]
        target_label: 'role'
      - source_labels: [__meta_consul_metadata_hostname]
        target_label: 'hostname'
      - target_label: '__scrape_interval__'
        replacement: '30s'  # 生产环境抓取更频繁

  # ============ 5. 按环境分组：测试环境 ============
  - job_name: 'huawei-ecs-test'
    consul_sd_configs:
      - server: '192.168.1.10:8500'
        services: ['node_exporter']
        token: '87654321-4321-4321-4321-cba987654321'
        tags: ['test']

    relabel_configs:
      - source_labels: [__meta_consul_health]
        regex: 'passing'
        action: keep
      - source_labels: [__meta_consul_address, __meta_consul_service_port]
        separator: ':'
        target_label: '__address__'
      - target_label: '__scrape_interval__'
        replacement: '60s'  # 测试环境可以抓取频率低一些

  # ============ 6. 按角色分组：数据库服务器 ============
  - job_name: 'huawei-ecs-database'
    consul_sd_configs:
      - server: '192.168.1.10:8500'
        services: ['node_exporter']
        token: '87654321-4321-4321-4321-cba987654321'

    relabel_configs:
      - source_labels: [__meta_consul_health]
        regex: 'passing'
        action: keep
      - source_labels: [__meta_consul_metadata_role]
        regex: 'db|database|mysql|redis'
        action: keep  # 只保留数据库相关角色
      - source_labels: [__meta_consul_address, __meta_consul_service_port]
        separator: ':'
        target_label: '__address__'

  # ============ 7. 按业务线分组：电商业务 ============
  - job_name: 'huawei-ecs-ecommerce'
    consul_sd_configs:
      - server: '192.168.1.10:8500'
        services: ['node_exporter']
        token: '87654321-4321-4321-4321-cba987654321'

    relabel_configs:
      - source_labels: [__meta_consul_health]
        regex: 'passing'
        action: keep
      - source_labels: [__meta_consul_metadata_business]
        regex: 'ecommerce'
        action: keep
      - source_labels: [__meta_consul_address, __meta_consul_service_port]
        separator: ':'
        target_label: '__address__'

3.2 按组查询的PromQL示例

完成分组配置后，你可以使用以下PromQL实现按组查询 ：

promql

# 1. 按环境统计节点数量
count by (env) (up{job="huawei-ecs-all"})

# 2. 按角色统计CPU使用率
avg by (role) (
  100 - (avg by (role, instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
)

# 3. 按业务线查看内存使用
sum by (business) (
  node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes
) / 1024 / 1024 / 1024

# 4. 查看生产环境各角色的磁盘使用率
avg by (role, instance) (
  (node_filesystem_size_bytes{fstype!="tmpfs"} - node_filesystem_free_bytes) 
  / node_filesystem_size_bytes{fstype!="tmpfs"} * 100
) and on(instance) (up{env="prod"} == 1)

# 5. 按团队聚合平均负载
avg by (team) (load1)

# 6. 多标签分组：按环境和角色
avg by (env, role) (
  100 - (avg by (env, role, instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
)

# 7. 使用group()进行标签分组（保留指定标签，丢弃其他）[citation:10]
group(node_cpu_seconds_total) by (env, role)

# 8. 复杂分组：按环境和实例类型
sum by (env, instance_type) (
  count by (env, instance_type, instance) (up)
)

# 9. 查看测试节点的具体指标
node_load1{env="test"}

# 10. 使用正则表达式过滤多个角色
up{role=~"web|api|frontend"}

3.3 Grafana自定义查询

在Grafana中添加Prometheus数据源后，创建Dashboard时可以使用变量实现动态分组：

Dashboard变量配置：

json

{
  "templating": {
    "list": [
      {
        "name": "env",
        "type": "query",
        "query": "label_values(up, env)"
      },
      {
        "name": "role",
        "type": "query",
        "query": "label_values(up{env=\"$env\"}, role)"
      },
      {
        "name": "instance",
        "type": "query",
        "query": "label_values(up{env=\"$env\", role=\"$role\"}, instance)"
      }
    ]
  }
}

常用Grafana面板查询：

promql

# CPU使用率（带变量）
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle", env="$env", role=~"$role"}[5m])) * 100)

# 内存使用率
(1 - node_memory_MemAvailable_bytes{env="$env"} / node_memory_MemTotal_bytes{env="$env"}) * 100

# 磁盘使用率TOP 10
topk(10, 
  (node_filesystem_size_bytes{fstype!="tmpfs"} - node_filesystem_free_bytes) 
  / node_filesystem_size_bytes{fstype!="tmpfs"} * 100
) by (instance, mountpoint)

⚠️ 第四阶段：告警规则配置

4.1 创建告警规则文件

创建 /opt/prometheus/alerts/ecs-alerts.yml ：

yaml

groups:
  - name: huawei_ecs_basic_alerts
    interval: 30s
    rules:
      # 实例存活告警
      - alert: ECSInstanceDown
        expr: up{job="huawei-ecs-all"} == 0
        for: 5m
        labels:
          severity: critical
          team: "{{ $labels.team }}"
        annotations:
          summary: "ECS实例 {{ $labels.hostname }} 已离线"
          description: "ECS实例 {{ $labels.hostname }} (环境: {{ $labels.env }}, 角色: {{ $labels.role }}) 已经离线超过5分钟。"

  - name: huawei_ecs_resource_alerts
    interval: 30s
    rules:
      # CPU使用率告警
      - alert: HighCPUUsage
        expr: 100 - (avg by (instance, env, role, team) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.env }}环境 {{ $labels.role }} CPU使用率过高"
          description: "{{ $labels.hostname }} CPU使用率已超过80% (当前值: {{ $value }}%)"

      # 内存使用率告警
      - alert: HighMemoryUsage
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "内存使用率过高 - {{ $labels.hostname }}"
          description: "{{ $labels.hostname }} 内存使用率已超过90% (当前值: {{ $value }}%)"

      # 磁盘使用率告警
      - alert: DiskSpaceLow
        expr: (node_filesystem_size_bytes{fstype!="tmpfs"} - node_filesystem_free_bytes) / node_filesystem_size_bytes{fstype!="tmpfs"} * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "磁盘空间不足 - {{ $labels.hostname }}"
          description: "{{ $labels.hostname }} 磁盘 {{ $labels.device }} 使用率已超过85%"

  - name: huawei_ecs_group_alerts
    interval: 30s
    rules:
      # 按环境分组告警：生产环境CPU过高
      - alert: ProdHighCPU
        expr: avg by (env) (100 - (avg by (env, instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)) > 70
        for: 10m
        labels:
          severity: warning
          env: prod
        annotations:
          summary: "生产环境整体CPU使用率过高"
          description: "生产环境整体CPU使用率已超过70% (当前值: {{ $value }}%)"

      # 按角色分组告警：数据库服务器磁盘不足
      - alert: DBDiskLow
        expr: avg by (role) ((node_filesystem_size_bytes{fstype!="tmpfs"} - node_filesystem_free_bytes) / node_filesystem_size_bytes{fstype!="tmpfs"} * 100) > 80
        for: 5m
        labels:
          severity: critical
          role: db
        annotations:
          summary: "数据库服务器磁盘空间即将耗尽"
          description: "数据库服务器平均磁盘使用率已超过80%"

      # 按业务线分组告警：电商业务负载过高
      - alert: EcommerceHighLoad
        expr: avg by (business) (load1) > 10
        for: 10m
        labels:
          severity: warning
          business: ecommerce
        annotations:
          summary: "电商业务线负载过高"
          description: "电商业务线平均负载已超过10 (当前值: {{ $value }})"

4.2 配置Alertmanager告警路由

更新 /opt/alertmanager/alertmanager.yml：

yaml

global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: 'alertmanager@example.com'
  smtp_auth_username: 'alertmanager@example.com'
  smtp_auth_password: 'password'

route:
  group_by: ['alertname', 'env', 'role', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  
  # 根据标签路由到不同接收器
  routes:
    # 严重告警发送到企业微信+邮件
    - match:
        severity: critical
      receiver: wechat-critical
      continue: true
    
    # 生产环境告警发送到邮件组
    - match:
        env: prod
      receiver: email-prod
      continue: true
    
    # 数据库相关告警发送到DBA团队
    - match:
        role: db
      receiver: email-dba
    
    # 默认路由
    - receiver: default

receivers:
  - name: 'default'
    email_configs:
      - to: 'ops@example.com'
        send_resolved: true

  - name: 'email-prod'
    email_configs:
      - to: 'prod-team@example.com'
        send_resolved: true

  - name: 'email-dba'
    email_configs:
      - to: 'dba-team@example.com'
        send_resolved: true

  - name: 'wechat-critical'
    webhook_configs:
      - url: 'http://wechat-bot.example.com/send'
        send_resolved: true

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'env', 'role']

4.3 重载配置

bash

# 检查Prometheus配置
/opt/prometheus/promtool check config /opt/prometheus/prometheus.yml

# 热加载
curl -X POST http://localhost:9090/-/reload

# 检查Alertmanager配置
/opt/alertmanager/amtool check-config /opt/alertmanager/alertmanager.yml

# 热加载
curl -X POST http://localhost:9093/-/reload

📈 第五阶段：扩展到全量节点

5.1 创建分组配置文件模板

bash

# 创建模板目录
mkdir -p /tmp/group-templates

# Web生产环境
cat > /tmp/group-templates/web-prod.conf <<EOF
ENVIRONMENT=prod
ROLE=web
BUSINESS=ecommerce
PROJECT=shop-frontend
OWNER=frontend-team
EOF

# Web预发布环境
cat > /tmp/group-templates/web-staging.conf <<EOF
ENVIRONMENT=staging
ROLE=web
BUSINESS=ecommerce
PROJECT=shop-frontend
OWNER=frontend-team
EOF

# 数据库生产环境
cat > /tmp/group-templates/db-prod.conf <<EOF
ENVIRONMENT=prod
ROLE=db
BUSINESS=ecommerce
PROJECT=shop-database
OWNER=dba-team
EOF

# Redis生产环境
cat > /tmp/group-templates/redis-prod.conf <<EOF
ENVIRONMENT=prod
ROLE=redis
BUSINESS=ecommerce
PROJECT=shop-cache
OWNER=backend-team
EOF

# 默认配置
cat > /tmp/group-templates/default.conf <<EOF
ENVIRONMENT=prod
ROLE=web
BUSINESS=default
PROJECT=default
OWNER=ops-team
EOF

5.2 增强版注册脚本

bash

cat > /tmp/register-to-consul-v2.sh <<'EOF'
#!/bin/bash
CONSUL_SERVER="192.168.1.10:8500"
TOKEN="87654321-4321-4321-4321-cba987654321"

# 获取基础信息
PRIVATE_IP=$(hostname -I | awk '{print $1}')
HOSTNAME=$(hostname)

# 从配置文件读取分组信息
GROUP_CONF="/etc/server-group.conf"
if [ -f "$GROUP_CONF" ]; then
    source "$GROUP_CONF"
else
    ENVIRONMENT="prod"
    ROLE="web"
    BUSINESS="default"
    PROJECT="default"
    OWNER="ops-team"
fi

# 构造标签
TAGS=("huaweicloud" "$ENVIRONMENT" "$ROLE" "$BUSINESS")

# 构造服务定义
cat > /tmp/service.json <<SERVICE_EOF
{
  "ID": "node-exporter-$HOSTNAME",
  "Name": "node_exporter",
  "Tags": ["${TAGS[0]}", "${TAGS[1]}", "${TAGS[2]}", "${TAGS[3]}"],
  "Address": "$PRIVATE_IP",
  "Port": 9100,
  "Meta": {
    "hostname": "$HOSTNAME",
    "environment": "$ENVIRONMENT",
    "role": "$ROLE",
    "business": "$BUSINESS",
    "project": "$PROJECT",
    "owner": "$OWNER",
    "registered_at": "$(date +%Y%m%d-%H%M%S)"
  },
  "Check": {
    "http": "http://$PRIVATE_IP:9100/metrics",
    "interval": "30s",
    "timeout": "5s",
    "deregister_critical_service_after": "10m"
  }
}
SERVICE_EOF

# 注册到Consul
curl -X PUT \
  -H "X-Consul-Token: $TOKEN" \
  --data @/tmp/service.json \
  http://$CONSUL_SERVER/v1/agent/service/register

echo "$(date): 注册 $HOSTNAME ($PRIVATE_IP) [$ENVIRONMENT/$ROLE/$BUSINESS]"
EOF

chmod +x /tmp/register-to-consul-v2.sh

5.3 使用clustershell批量部署到全量节点

bash

# 1. 安装Node Exporter到所有节点
clush -a "
# 下载安装node_exporter
wget -q https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xf node_exporter-1.7.0.linux-amd64.tar.gz
sudo mv node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/

# 创建systemd服务
sudo tee /etc/systemd/system/node_exporter.service <<'SERVICE_EOF'
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=nobody
Group=nogroup
Type=simple
ExecStart=/usr/local/bin/node_exporter --web.listen-address=:9100
Restart=always

[Install]
WantedBy=multi-user.target
SERVICE_EOF

# 启动
sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
"

# 2. 根据节点角色分发分组配置文件
# Web生产环境
clush -g web-prod -c /tmp/group-templates/web-prod.conf --dest=/etc/server-group.conf
# Web预发布环境
clush -g web-staging -c /tmp/group-templates/web-staging.conf --dest=/etc/server-group.conf
# DB生产环境
clush -g db-prod -c /tmp/group-templates/db-prod.conf --dest=/etc/server-group.conf
# Redis生产环境
clush -g redis-prod -c /tmp/group-templates/redis-prod.conf --dest=/etc/server-group.conf
# 其他节点使用默认配置
clush -g others -c /tmp/group-templates/default.conf --dest=/etc/server-group.conf

# 3. 分发注册脚本
clush -a -c /tmp/register-to-consul-v2.sh --dest=/usr/local/bin/
clush -a "chmod +x /usr/local/bin/register-to-consul-v2.sh"

# 4. 执行首次注册
clush -a "/usr/local/bin/register-to-consul-v2.sh"

# 5. 添加到crontab
clush -a 'echo "*/5 * * * * root /usr/local/bin/register-to-consul-v2.sh" >> /etc/crontab'

🔍 第六阶段：验证与测试

6.1 验证Consul注册

bash

# 查看所有服务
curl -H "X-Consul-Token: $CONSUL_TOKEN" http://localhost:8500/v1/catalog/services

# 按环境统计
curl -s -H "X-Consul-Token: $CONSUL_TOKEN" \
  "http://localhost:8500/v1/health/service/node_exporter?passing" | \
  jq -r '.[].Service.Meta.environment' | sort | uniq -c

# 按角色统计
curl -s -H "X-Consul-Token: $CONSUL_TOKEN" \
  "http://localhost:8500/v1/health/service/node_exporter?passing" | \
  jq -r '.[].Service.Meta.role' | sort | uniq -c

# 按业务线统计
curl -s -H "X-Consul-Token: $CONSUL_TOKEN" \
  "http://localhost:8500/v1/health/service/node_exporter?passing" | \
  jq -r '.[].Service.Meta.business' | sort | uniq -c

6.2 验证Prometheus Targets

bash

# 查看所有targets
curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, instance: .labels.instance, health: .health}'

# 按job统计
curl -s http://localhost:9090/api/v1/targets | \
  jq '.data.activeTargets | group_by(.labels.job) | map({job: .[0].labels.job, count: length})'

6.3 测试告警

bash

# 手动触发告警（停止一个Node Exporter）
clush -w 192.168.1.101 "sudo systemctl stop node_exporter"

# 等待5分钟后查看告警
curl http://localhost:9093/api/v2/alerts | jq '.[] | {name: .labels.alertname, status: .status.state}'

# 恢复
clush -w 192.168.1.101 "sudo systemctl start node_exporter"

6.4 Grafana导入Dashboard

bash

# 推荐导入的Dashboard ID
# 8919 - Node Exporter Server Metrics
# 11074 - Node Exporter 1-Panel
# 1860 - Node Exporter Full

# 或者创建自定义Dashboard，使用以下PromQL

promql

# 自定义分组面板
# 按环境分组CPU使用率
avg by (env) (100 - (avg by (env, instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100))

# 按角色分组内存使用
sum by (role) (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / 1024 / 1024 / 1024

# 按业务线分组节点数量
count by (business) (up)

# 按团队分组磁盘总量
sum by (team) (node_filesystem_size_bytes{fstype!="tmpfs"}) / 1024 / 1024 / 1024 / 1024

📋 常见问题解答

Q1: Node Exporter启动方式选择？

A:

  • systemd方式：适合生产环境，稳定可靠，有完善的日志和重启策略

  • Docker方式：适合测试环境或容器化部署，但需要注意网络模式设置 

Q2: 克隆机器是否有限制？

A: Node Exporter本身没有client ID限制。只要确保每台机器的：

  • 主机名唯一（用于生成Service ID）

  • IP地址唯一

  • 注册到Consul时使用唯一的Service ID（我们使用node-exporter-$(hostname)确保唯一性）

Q3: 如何实现按组查询？

A: 通过Consul注册时的标签（Tags）和元数据（Meta），Prometheus会自动将这些信息转换为标签，然后使用PromQL的by子句进行分组查询 ：

promql

# 按环境分组
avg by (env) (rate(node_cpu_seconds_total[5m]))

# 多标签分组
sum by (env, role) (node_memory_MemTotal_bytes)

Q4: 告警规则如何按分组路由？

A: 在Alertmanager配置中，可以根据告警的标签进行路由 ：

yaml

routes:
  - match:
      env: prod
      severity: critical
    receiver: prod-critical

Q5: 如何测试告警是否正常工作？

A: 可以：

  1. 手动停止一个Node Exporter服务

  2. 查看Prometheus Alerts页面确认告警状态变为PENDING→FIRING

  3. 查看Alertmanager接收到的告警

  4. 验证通知渠道是否收到消息

🎯 总结

已完成的功能

阶段 功能 状态
第一阶段 Consul集群、Prometheus、Grafana、Alertmanager基础环境 ✅
第二阶段 3台测试节点部署，验证动态发现 ✅
第三阶段 Prometheus分组配置，按环境/角色/业务线分组 ✅
第四阶段 告警规则配置，按分组路由 ✅
第五阶段 扩展到全量节点，分组模板 ✅
第六阶段 验证与测试，自定义Grafana查询 ✅

可扩展的方向

  1. 多集群支持：配置多个Consul datacenter

  2. 自定义Exporter：MySQL、Nginx、Redis等应用监控

  3. 日志集成：添加Loki实现日志与指标联动

  4. 自动化运维：通过Consul API实现自动扩缩容时的服务注册/注销

# prometheus # consul
Logo
https://edu.csdn.net/learn/39067/627173?utm_source=2019755004

汇聚全球AI编程工具，助力开发者即刻编程。

更多推荐

cover

一个 API 入口调用多个大模型：AiiOnly客户端 + CC Switch 打通 Codex AI编程流程

avatar AI编程社区
cover

从零开始做一个属于自己的 Skill（保姆级教程）

avatar AI编程社区
cover

VSCode Claude Code + MiMo 中转免登录完整配置教程（解决原生登录弹窗 + 模型报错）

avatar AI编程社区