概述

本文介绍了在Ubuntu系统上部署NVIDIA GPU监控告警系统的完整流程。主要内容包括：1）通过DCGM Exporter监控GPU指标；2）配置Prometheus收集主机和GPU数据；3）设置Alertmanager实现邮件告警，包含CPU、内存、磁盘和GPU温度/显存等告警规则；4）使用Grafana进行可视化展示。系统采用Docker容器化部署，实现了对服务器资源和GPU状态的全面监控与告警功能

前提条件：安装了NVIDIA驱动的Linux服务器

Ubantu系统

step1 部署DCGM Exporter

# 添加NVIDIA仓库
distribution=$(. /etc/os-release;echo $ID$VERSION_ID | sed -e 's/\.//g')
wget https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/cuda-keyring_1.0-1_all.deb
sudo dpkg -i cuda-keyring_1.0-1_all.deb
​
# 更新并安装DCGM
sudo apt-get update
sudo apt-get install -y datacenter-gpu-manager
​
# 启动DGGM服务
sudo systemctl start nvidia-dcgm
sudo systemctl enable nvidia-dcgm
# 验证DGGM安装
dcgmi discovery -l


# 使用Docker部署DCGM Exporter
docker run -d --gpus all --restart always \
  -p 9400:9400 \
  --name dcgm-exporter \
  nvcr.io/nvidia/k8s/dcgm-exporter:3.3.6-3.4.2-ubuntu22.04


# 部署node-exporter监控cpu/内存/IO/网络
docker run -d -p 9100:9100 --name node-exporter prom/node-exporter 

step2 编写告警配置文件

$ mkdir -p /opt/alertmanager/config
$ vim /opt/alertmanager/config/alertmanager.yml
global:
  smtp_smarthost: 'smtp.163.com:465'  
  smtp_from: '邮箱地址'    
  smtp_auth_username: '邮箱地址'  
  smtp_auth_password: '授权码'       
  smtp_require_tls: false               
  smtp_hello: '163.com'                
​
​
route:
  group_by: ['alertname']  
  group_wait: 10s          
  group_interval: 10s      
  repeat_interval: 1h      
  receiver: '163-email'    
templates:
  - '/opt/alertmanager/config/*.tmpl'
​
receivers:
- name: '163-email'  
  email_configs:
  - to: '2831205002@qq.com' 
    send_resolved: true          
    headers:
      Subject: "【监控告警】{{ .CommonAnnotations.summary }}"

 

 

step3 编写邮箱模板

{{ define "email.html" }}
{{ range .Alerts }}
告警程序: prometheus_alert <br>
告警级别: {{ .Labels.severity }} 级 <br>
告警类型: {{ .Labels.alertname }} <br>
故障主机: {{ .Labels.instance }} <br>
告警主题: {{ .Annotations.summary }} <br>
告警详情: {{ .Annotations.description }} <br>
触发时间: {{ (.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }} <br>
{{ end }}
{{ end }}

​

step4 创建网络

# 创建网络，将altermanager和prometheus加入该网络
docker create network monitor-network
​
# 启动alertmanager
docker run -d \
  --name alertmanager \ 
  --network monitor-network \  # 加入监控网络，与Prometheus互通
  -p 9093:9093 \
  -v /opt/alertmanager/config/alertmanager.yml:/etc/alertmanager/alertmanager.yml \ 
  --restart=always \ 
  prom/alertmanager

 

step5 编写prometheus配置文件

#创建配置文件/opt/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'dcgm-exporter'
    static_configs:
      - targets: ['监测服务器IP:9400']

  - job_name: 'prometheus'
    static_configs:
      - targets: ['监测服务器IP:9090']

  - job_name: 'node'
    static_configs:
      - targets: ['监测服务器IP:9100']

alerting:
  alertmanagers:
  - static_configs:
      - targets: ['监测服务器IP:9093']

rule_files:
  - "/etc/prometheus/rules/*.yml"   #容器报警规则存放路径

step5 编写CPU/内存/磁盘告警规则

$ vim /opt/prometheus/rules/node_alter.yml

groups:
- name: 实例存活告警规则
  rules:
  - alert: 实例存活告警
    expr: up == 0
    for: 30s
    labels:
      severity: Disaster
    annotations:
      summary: "节点失联"
      description: "节点断联已超过1分钟"
- name: 内存告警规则
  rules:
  - alert: "内存使用率告警"
    expr: (node_memory_MemTotal_bytes - (node_memory_MemFree_bytes+node_memory_Buffers_bytes+node_memory_Cached_bytes )) / node_memory_MemTotal_bytes * 100 > 75
    for: 30s
    labels:
      severity: warning
    annotations:
      summary: "服务器内存报警"
      description: "内存资源利用率大于75%！(当前值: {{ $value }}%)"
- name: 磁盘告警规则
  rules:
  - alert: 磁盘使用率告警
    expr: (node_filesystem_size_bytes - node_filesystem_avail_bytes) / node_filesystem_size_bytes * 100 > 80
    for: 30s
    labels:
      severity: warning
    annotations:
      summary: "服务器磁盘报警"
      description: "服务器磁盘设备使用超过80%！(挂载点: {{ $labels.mountpoint }} 当前值: {{ $value }}%)"

 

step6 编写监控 GPU报警规则

$ vim /opt/prometheus/rules/gpu_alerts.yml 

groups:
- name: gpu_alerts
  rules:
    # GPU温度过高
    - alert: GPU温度
      expr: avg by (gpu_id) (nvidia_gpu_temperature_celsius) > 85
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "GPU温度过高 (实例: {{ $labels.gpu_id }})"
        description: "GPU {{ $labels.gpu_id }} 温度持续5分钟高于85°C，当前值: {{ $value }}°C"

    # GPU显存使用率超过阈值
    - alert: GPU显存
      expr: (nvidia_gpu_memory_used_bytes / nvidia_gpu_memory_total_bytes) * 100 > 90
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "GPU显存不足 (实例: {{ $labels.gpu_id }})"
        description: "GPU {{ $labels.gpu_id }} 显存使用率超过90%，当前值: {{ $value | humanize }}%"

    # GPU进程异常(CURD错误)
    - alert: CRUD进程
      expr: changes(nvidia_gpu_cuda_process_count[5m]) < 0
      for: 0m
      labels:
        severity: critical
      annotations:
        summary: "GPU进程异常终止 (实例: {{ $labels.gpu_id }})"
        description: "GPU {{ $labels.gpu_id }} 上的CUDA进程数量减少，可能发生崩溃"




#使用Docker部署Prometheus：
docker run -d \
  --name prometheus \
  --network monitor-network \
  -p 9090:9090 \
  -v /opt/prometheus/config/prometheus.yml:/etc/prometheus/prometheus.yml \
  -v /opt/prometheus/rules:/etc/prometheus/rules \  
  --restart=always \
  prom/prometheus

验证Prometheus是否正常工作：访问http://IP:9090

检查目标是否正常：在Prometheus界面导航到Status > Targets，确认dcgm-exporter目标状态为UP

 

step7 服务器搭建Grafana

#使用Docker部署Grafana
docker run -d --restart always \
  -p 3000:3000 \
  --name grafana \
  grafana/grafana

访问Grafana：打开浏览器访问http://IP:3000，用户名：admin` 密码：自定义

 

导入数据源

 

导入面板

Grafana官方提供模板地址：https://grafana.com/grafana/dashboards