YOLO12模型监控体系：Prometheus+Grafana实时跟踪GPU利用率与QPS
·
YOLO12模型监控体系：Prometheus+Grafana实时跟踪GPU利用率与QPS
部署一个高性能的YOLO12目标检测模型只是第一步。当模型真正投入生产环境，面对7x24小时不间断的视频流处理时，如何确保它稳定运行？如何知道GPU资源是否被充分利用？如何及时发现性能瓶颈？
很多开发者都有这样的经历：模型在测试时表现良好，一旦上线就出现各种问题——GPU利用率忽高忽低、推理速度时快时慢、内存泄漏导致服务崩溃。等到用户投诉才发现问题，已经造成了业务损失。
本文将带你搭建一套完整的YOLO12模型监控体系，使用Prometheus+Grafana实时跟踪GPU利用率、显存占用、QPS（每秒查询数）等关键指标。这套方案不仅能让你随时掌握模型运行状态，还能通过历史数据分析性能趋势，为容量规划和优化提供数据支撑。
1. 为什么YOLO12需要专门的监控体系
YOLO12作为实时目标检测模型，在生产环境中面临几个独特的挑战：
实时性要求高：安防监控、自动驾驶等场景需要毫秒级响应，任何性能下降都会直接影响业务效果。
资源消耗波动大：不同场景的图像复杂度差异巨大——简单场景可能只检测几个人，复杂场景可能需要同时识别几十个不同类别的物体，GPU利用率可能从30%瞬间飙升到90%。
多实例部署常见：为了处理高并发请求，通常会部署多个YOLO12实例。如何平衡各个实例的负载？如何知道该扩容还是缩容？
故障难以复现：很多性能问题只在特定条件下出现，如果没有实时监控和历史数据，很难定位根本原因。
传统的日志监控只能告诉你“服务挂了”，但无法回答“为什么挂”、“挂之前发生了什么”。我们需要更细粒度的监控，能够实时看到：
- GPU到底有多忙？
- 显存用了多少？会不会泄漏？
- 每秒处理多少张图片？
- 平均响应时间是多少？
- 错误率有多高？
2. 监控方案整体架构
我们的监控体系基于云原生监控的黄金标准组合：Prometheus + Grafana。
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ │ │ │ │ │
│ YOLO12服务 │────▶│ Prometheus │────▶│ Grafana │
│ (FastAPI) │ │ (指标收集) │ │ (可视化) │
│ │ │ │ │ │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
│ 自定义指标 │ 定时抓取 │ 实时展示
│ (GPU/QPS/延迟) │ (15秒间隔) │ (仪表盘)
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ │ │ │ │ │
│ NVIDIA DCGM │ │ Node Exporter │ │ Alertmanager │
│ (GPU监控) │ │ (系统监控) │ │ (告警) │
│ │ │ │ │ │
└─────────────────┘ └─────────────────┘ └─────────────────┘
各组件分工：
- YOLO12服务：通过FastAPI中间件收集推理指标（QPS、延迟、错误率）
- NVIDIA DCGM：采集GPU硬件指标（利用率、显存、温度）
- Node Exporter：采集系统指标（CPU、内存、磁盘、网络）
- Prometheus：定时抓取所有指标并存储到时序数据库
- Grafana：通过丰富的图表展示监控数据
- Alertmanager：根据规则发送告警（可选）
这套架构的优点是：
- 开源免费：所有组件都是开源软件
- 扩展性强：可以轻松添加新的监控指标
- 可视化好：Grafana的仪表盘非常直观
- 历史数据：可以查看任意时间段的性能趋势
3. 环境准备与组件安装
3.1 基础环境检查
首先确保你的YOLO12服务运行环境满足以下要求：
# 检查Python版本
python --version
# Python 3.8+
# 检查PyTorch和CUDA
python -c "import torch; print(f'PyTorch: {torch.__version__}'); print(f'CUDA可用: {torch.cuda.is_available()}')"
# 检查GPU信息
nvidia-smi
如果你的YOLO12是基于我们提供的镜像部署的，那么环境已经预装了CUDA 12.4和PyTorch 2.5.0。
3.2 安装监控组件
我们将使用Docker来部署监控组件，这样最方便也最干净。
# 创建监控专用目录
mkdir -p ~/yolo12-monitoring
cd ~/yolo12-monitoring
# 创建docker-compose.yml文件
cat > docker-compose.yml << 'EOF'
version: '3.8'
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
restart: unless-stopped
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/console_templates'
- '--storage.tsdb.retention.time=30d'
- '--web.enable-lifecycle'
ports:
- "9090:9090"
networks:
- monitoring
grafana:
image: grafana/grafana:latest
container_name: grafana
restart: unless-stopped
volumes:
- grafana_data:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin123
- GF_USERS_ALLOW_SIGN_UP=false
ports:
- "3000:3000"
networks:
- monitoring
node-exporter:
image: prom/node-exporter:latest
container_name: node-exporter
restart: unless-stopped
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/rootfs:ro
command:
- '--path.procfs=/host/proc'
- '--path.rootfs=/rootfs'
- '--path.sysfs=/host/sys'
- '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
ports:
- "9100:9100"
networks:
- monitoring
nvidia-dcgm-exporter:
image: nvidia/dcgm-exporter:latest
container_name: nvidia-dcgm-exporter
restart: unless-stopped
environment:
- NVIDIA_VISIBLE_DEVICES=all
volumes:
- /run/nvidia:/run/nvidia
ports:
- "9400:9400"
networks:
- monitoring
networks:
monitoring:
driver: bridge
volumes:
prometheus_data:
grafana_data:
EOF
3.3 配置Prometheus
创建Prometheus的配置文件，告诉它要监控哪些目标：
# 创建prometheus.yml配置文件
cat > prometheus.yml << 'EOF'
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'node-exporter'
static_configs:
- targets: ['node-exporter:9100']
- job_name: 'nvidia-gpu'
static_configs:
- targets: ['nvidia-dcgm-exporter:9400']
- job_name: 'yolo12-api'
static_configs:
- targets: ['host.docker.internal:8000'] # 你的YOLO12 API地址
metrics_path: '/metrics'
scrape_interval: 10s # YOLO12指标变化快，缩短采集间隔
EOF
重要提示：如果你的YOLO12服务运行在Docker容器外（比如直接在宿主机上），需要将host.docker.internal改为宿主机的实际IP地址。
3.4 启动监控服务
# 启动所有监控组件
docker-compose up -d
# 检查服务状态
docker-compose ps
# 查看日志（如果有问题）
docker-compose logs -f prometheus
服务启动后，可以通过以下地址访问：
- Prometheus: http://localhost:9090
- Grafana: http://localhost:3000 (用户名: admin, 密码: admin123)
- Node Exporter指标: http://localhost:9100/metrics
- NVIDIA GPU指标: http://localhost:9400/metrics
4. 为YOLO12服务添加指标暴露
现在监控基础设施已经就绪，接下来需要让YOLO12服务暴露自己的性能指标。
4.1 安装Prometheus Python客户端
首先在YOLO12服务环境中安装必要的库：
pip install prometheus-client psutil
4.2 创建指标收集中间件
在YOLO12的FastAPI应用中添加一个中间件，用于收集推理相关的指标：
# metrics_middleware.py
import time
from prometheus_client import Counter, Gauge, Histogram, generate_latest, REGISTRY
from prometheus_client.openmetrics.exposition import CONTENT_TYPE_LATEST
from fastapi import Request, Response
import psutil
import torch
# 定义Prometheus指标
# 请求相关指标
REQUEST_COUNT = Counter(
'yolo12_requests_total',
'Total number of requests',
['method', 'endpoint', 'status']
)
REQUEST_LATENCY = Histogram(
'yolo12_request_duration_seconds',
'Request latency in seconds',
['method', 'endpoint']
)
# 推理相关指标
INFERENCE_COUNT = Counter(
'yolo12_inferences_total',
'Total number of inferences'
)
INFERENCE_LATENCY = Histogram(
'yolo12_inference_duration_seconds',
'Inference latency in seconds',
['model_size'] # 区分不同模型大小
)
# GPU相关指标
GPU_UTILIZATION = Gauge(
'yolo12_gpu_utilization_percent',
'GPU utilization percentage',
['gpu_id']
)
GPU_MEMORY_USED = Gauge(
'yolo12_gpu_memory_used_mb',
'GPU memory used in MB',
['gpu_id']
)
GPU_MEMORY_TOTAL = Gauge(
'yolo12_gpu_memory_total_mb',
'GPU total memory in MB',
['gpu_id']
)
# QPS指标
QPS_GAUGE = Gauge(
'yolo12_qps',
'Queries per second'
)
# 系统资源指标
CPU_USAGE = Gauge(
'yolo12_cpu_usage_percent',
'CPU usage percentage'
)
MEMORY_USED = Gauge(
'yolo12_memory_used_mb',
'Memory used in MB'
)
MEMORY_TOTAL = Gauge(
'yolo12_memory_total_mb',
'Total memory in MB'
)
class MetricsMiddleware:
"""Prometheus指标收集中间件"""
def __init__(self, app):
self.app = app
self.last_update_time = time.time()
self.request_count_last_minute = 0
async def __call__(self, scope, receive, send):
if scope['type'] != 'http':
await self.app(scope, receive, send)
return
request = Request(scope, receive)
method = request.method
endpoint = request.url.path
# 记录请求开始时间
start_time = time.time()
# 定义发送响应的包装函数
async def send_wrapper(message):
if message['type'] == 'http.response.start':
status_code = message['status']
# 记录请求指标
REQUEST_COUNT.labels(
method=method,
endpoint=endpoint,
status=status_code
).inc()
# 记录延迟
latency = time.time() - start_time
REQUEST_LATENCY.labels(
method=method,
endpoint=endpoint
).observe(latency)
# 如果是推理请求，记录推理指标
if endpoint == '/predict':
INFERENCE_COUNT.inc()
INFERENCE_LATENCY.labels(
model_size=os.environ.get('YOLO_MODEL', 'yolov12n.pt')
).observe(latency)
# 更新QPS
self.request_count_last_minute += 1
current_time = time.time()
time_diff = current_time - self.last_update_time
if time_diff >= 1.0: # 每秒更新一次QPS
qps = self.request_count_last_minute / time_diff
QPS_GAUGE.set(qps)
self.request_count_last_minute = 0
self.last_update_time = current_time
await send(message)
await self.app(scope, receive, send_wrapper)
def update_system_metrics():
"""更新系统资源指标"""
# CPU使用率
cpu_percent = psutil.cpu_percent(interval=1)
CPU_USAGE.set(cpu_percent)
# 内存使用
memory = psutil.virtual_memory()
MEMORY_USED.set(memory.used / 1024 / 1024) # 转换为MB
MEMORY_TOTAL.set(memory.total / 1024 / 1024) # 转换为MB
# GPU指标（如果可用）
if torch.cuda.is_available():
for i in range(torch.cuda.device_count()):
gpu_util = torch.cuda.utilization(i)
gpu_mem = torch.cuda.memory_allocated(i)
gpu_mem_total = torch.cuda.get_device_properties(i).total_memory
GPU_UTILIZATION.labels(gpu_id=str(i)).set(gpu_util)
GPU_MEMORY_USED.labels(gpu_id=str(i)).set(gpu_mem / 1024 / 1024) # MB
GPU_MEMORY_TOTAL.labels(gpu_id=str(i)).set(gpu_mem_total / 1024 / 1024) # MB
def setup_metrics_endpoint(app):
"""设置/metrics端点"""
@app.get("/metrics")
async def metrics():
# 更新系统指标
update_system_metrics()
# 返回所有指标
return Response(
content=generate_latest(REGISTRY),
media_type=CONTENT_TYPE_LATEST
)
4.3 集成到YOLO12 FastAPI应用
修改你的YOLO12 FastAPI应用，集成指标收集：
# main.py (YOLO12 FastAPI应用)
import os
import time
from fastapi import FastAPI, File, UploadFile
from fastapi.responses import JSONResponse
import uvicorn
from prometheus_client import start_http_server
import threading
# 导入指标中间件
from metrics_middleware import MetricsMiddleware, setup_metrics_endpoint
app = FastAPI(title="YOLO12 Detection API")
# 添加指标中间件
app.add_middleware(MetricsMiddleware)
# 设置/metrics端点
setup_metrics_endpoint(app)
# 启动Prometheus客户端HTTP服务器（在另一个端口）
def start_metrics_server():
start_http_server(8001) # 在8001端口暴露指标
# 在后台线程中启动指标服务器
metrics_thread = threading.Thread(target=start_metrics_server, daemon=True)
metrics_thread.start()
@app.post("/predict")
async def predict(file: UploadFile = File(...)):
"""目标检测接口"""
start_time = time.time()
try:
# 读取图片
contents = await file.read()
# 这里是你原有的YOLO12推理代码
# results = model(contents)
# 模拟推理时间（实际使用时替换为真实推理）
import random
time.sleep(random.uniform(0.01, 0.05)) # 10-50ms模拟推理
# 模拟返回结果
results = {
"detections": [
{
"bbox": [100, 100, 200, 200],
"confidence": 0.95,
"class": "person"
}
],
"inference_time": time.time() - start_time
}
return JSONResponse(content=results)
except Exception as e:
return JSONResponse(
status_code=500,
content={"error": str(e)}
)
if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=8000)
现在你的YOLO12服务会在两个端口提供服务：
8000端口：正常的API服务8001端口：Prometheus指标端点
更新Prometheus配置，添加对8001端口的监控：
# 在prometheus.yml中添加
- job_name: 'yolo12-metrics'
static_configs:
- targets: ['host.docker.internal:8001'] # YOLO12指标端点
scrape_interval: 5s # 更短的间隔，实时性要求高
5. 配置Grafana监控仪表盘
现在所有指标都已经收集到Prometheus中，接下来在Grafana中创建监控仪表盘。
5.1 添加Prometheus数据源
- 访问 http://localhost:3000，使用admin/admin123登录
- 点击左侧齿轮图标 → Data Sources → Add data source
- 选择 Prometheus
- 配置URL为 http://prometheus:9090
- 点击 Save & Test，应该显示"Data source is working"
5.2 导入YOLO12监控仪表盘
Grafana社区有很多现成的仪表盘，但我们需要一个专门为YOLO12定制的。这里我提供一个完整的仪表盘配置：
{
"dashboard": {
"title": "YOLO12模型监控仪表盘",
"panels": [
{
"title": "GPU利用率",
"targets": [{
"expr": "yolo12_gpu_utilization_percent",
"legendFormat": "GPU {{gpu_id}}"
}],
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 0}
},
{
"title": "GPU显存使用",
"targets": [
{"expr": "yolo12_gpu_memory_used_mb", "legendFormat": "已使用"},
{"expr": "yolo12_gpu_memory_total_mb", "legendFormat": "总量"}
],
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 0}
},
{
"title": "QPS（每秒查询数）",
"targets": [{
"expr": "yolo12_qps",
"legendFormat": "QPS"
}],
"type": "stat",
"gridPos": {"h": 6, "w": 6, "x": 0, "y": 8}
},
{
"title": "总推理次数",
"targets": [{
"expr": "increase(yolo12_inferences_total[1h])",
"legendFormat": "过去1小时"
}],
"type": "stat",
"gridPos": {"h": 6, "w": 6, "x": 6, "y": 8}
},
{
"title": "平均推理延迟",
"targets": [{
"expr": "rate(yolo12_inference_duration_seconds_sum[5m]) / rate(yolo12_inference_duration_seconds_count[5m])",
"legendFormat": "平均延迟"
}],
"type": "stat",
"gridPos": {"h": 6, "w": 6, "x": 12, "y": 8},
"fieldConfig": {
"defaults": {
"unit": "s",
"decimals": 3
}
}
},
{
"title": "P95推理延迟",
"targets": [{
"expr": "histogram_quantile(0.95, sum(rate(yolo12_inference_duration_seconds_bucket[5m])) by (le))",
"legendFormat": "P95延迟"
}],
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 14},
"fieldConfig": {
"defaults": {
"unit": "s",
"decimals": 3
}
}
},
{
"title": "请求成功率",
"targets": [{
"expr": "sum(rate(yolo12_requests_total{status=~\"2..\"}[5m])) / sum(rate(yolo12_requests_total[5m])) * 100",
"legendFormat": "成功率"
}],
"type": "gauge",
"gridPos": {"h": 6, "w": 6, "x": 12, "y": 8},
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100
}
}
},
{
"title": "CPU使用率",
"targets": [{
"expr": "yolo12_cpu_usage_percent",
"legendFormat": "CPU"
}],
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 22}
},
{
"title": "内存使用",
"targets": [
{"expr": "yolo12_memory_used_mb", "legendFormat": "已使用"},
{"expr": "yolo12_memory_total_mb", "legendFormat": "总量"}
],
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 22}
},
{
"title": "请求分布（最近1小时）",
"targets": [{
"expr": "sum by (endpoint) (rate(yolo12_requests_total[1h]))",
"legendFormat": "{{endpoint}}"
}],
"type": "piechart",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 30}
},
{
"title": "错误请求率",
"targets": [{
"expr": "sum(rate(yolo12_requests_total{status=~\"5..\"}[5m])) / sum(rate(yolo12_requests_total[5m])) * 100",
"legendFormat": "错误率"
}],
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 30},
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100
}
}
}
],
"time": {
"from": "now-1h",
"to": "now"
},
"refresh": "5s"
}
}
在Grafana中导入这个仪表盘：
- 点击左侧"+"图标 → Import
- 将上面的JSON粘贴到"Import via panel json"中
- 点击Load
- 选择Prometheus数据源
- 点击Import
5.3 关键指标解读
现在你的仪表盘上会显示以下关键指标：
GPU监控区域：
- GPU利用率：理想情况下应该保持在70-90%，太低说明资源浪费，太高可能影响稳定性
- GPU显存：注意观察是否有内存泄漏（持续增长不释放）
性能监控区域：
- QPS：每秒处理的图片数量，这是衡量服务吞吐量的核心指标
- 推理延迟：平均延迟和P95延迟，P95更能反映用户体验
- 总推理次数：了解服务负载情况
系统资源区域：
- CPU使用率：YOLO12主要用GPU，CPU使用率通常不高
- 内存使用：监控系统内存，防止OOM（内存溢出）
业务监控区域：
- 请求成功率：应该接近100%，低于99%需要关注
- 错误请求率：监控5xx错误，及时发现服务异常
- 请求分布：了解不同端点的调用频率
6. 实战：通过监控数据优化YOLO12性能
有了监控数据，我们就可以基于数据做优化决策。下面看几个实际场景：
6.1 场景一：GPU利用率低怎么办？
问题现象：GPU利用率长期低于30%，但QPS也不高。
可能原因：
- 请求量不足，GPU"吃不饱"
- 图片预处理（resize、归一化）在CPU进行，成为瓶颈
- 后处理（NMS、结果格式化）耗时太长
解决方案：
# 优化方案：使用GPU加速预处理
import torch
import cv2
import numpy as np
from torchvision import transforms
from PIL import Image
class OptimizedPreprocessor:
def __init__(self, device='cuda'):
self.device = device
# 使用GPU加速的预处理流水线
self.transform = transforms.Compose([
transforms.Resize((640, 640)),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
]).to(device)
def preprocess(self, image_bytes):
# 在CPU上快速解码
image = Image.open(io.BytesIO(image_bytes))
# 转换为tensor并移动到GPU
tensor = self.transform(image).unsqueeze(0).to(self.device)
return tensor
# 在推理时使用
preprocessor = OptimizedPreprocessor(device='cuda')
@app.post("/predict_optimized")
async def predict_optimized(file: UploadFile = File(...)):
contents = await file.read()
# GPU加速的预处理
input_tensor = preprocessor.preprocess(contents)
# 推理（已经在GPU上）
with torch.no_grad():
results = model(input_tensor)
# GPU上的后处理
processed_results = process_on_gpu(results)
# 只有最终结果才移回CPU
return processed_results.cpu().numpy()
6.2 场景二：QPS波动大怎么办？
问题现象：QPS时高时低，不稳定。
可能原因：
- 请求不均匀，有突发流量
- 某些图片特别复杂，处理时间长
- 有资源竞争（多个服务共享GPU）
解决方案：
# 添加请求队列和批处理
from queue import Queue
import threading
import time
class BatchProcessor:
def __init__(self, model, batch_size=8, max_wait_time=0.1):
self.model = model
self.batch_size = batch_size
self.max_wait_time = max_wait_time
self.queue = Queue()
self.results = {}
self.lock = threading.Lock()
self.thread = threading.Thread(target=self._process_batch, daemon=True)
self.thread.start()
def _process_batch(self):
while True:
batch = []
batch_ids = []
start_time = time.time()
# 收集一个batch或等待超时
while len(batch) < self.batch_size:
try:
# 最多等待max_wait_time秒
remaining = self.max_wait_time - (time.time() - start_time)
if remaining <= 0 and batch:
break
item_id, image_tensor = self.queue.get(timeout=remaining)
batch.append(image_tensor)
batch_ids.append(item_id)
except:
if batch:
break
time.sleep(0.01)
if batch:
# 批量推理
batch_tensor = torch.cat(batch, dim=0)
with torch.no_grad():
batch_results = self.model(batch_tensor)
# 分发结果
with self.lock:
for i, item_id in enumerate(batch_ids):
self.results[item_id] = batch_results[i]
def predict(self, image_tensor):
item_id = str(time.time()) + str(id(image_tensor))
self.queue.put((item_id, image_tensor))
# 等待结果
while True:
with self.lock:
if item_id in self.results:
result = self.results.pop(item_id)
return result
time.sleep(0.001)
# 使用批处理器
batch_processor = BatchProcessor(model, batch_size=8)
@app.post("/predict_batch")
async def predict_batch(file: UploadFile = File(...)):
contents = await file.read()
image_tensor = preprocessor.preprocess(contents)
result = batch_processor.predict(image_tensor)
return result
6.3 场景三：如何根据监控数据自动扩缩容？
基于监控数据，我们可以实现自动扩缩容策略：
# autoscaler.py
import time
import requests
import json
class YOLO12AutoScaler:
def __init__(self, prometheus_url="http://localhost:9090"):
self.prometheus_url = prometheus_url
self.scale_up_threshold = 80 # GPU利用率>80%时扩容
self.scale_down_threshold = 30 # GPU利用率<30%时缩容
self.min_instances = 1
self.max_instances = 10
self.current_instances = 1
def query_prometheus(self, query):
"""查询Prometheus指标"""
response = requests.get(
f"{self.prometheus_url}/api/v1/query",
params={'query': query}
)
return response.json()
def get_gpu_utilization(self):
"""获取平均GPU利用率"""
result = self.query_prometheus('avg(yolo12_gpu_utilization_percent)')
if result['status'] == 'success' and result['data']['result']:
return float(result['data']['result'][0]['value'][1])
return 0
def get_qps(self):
"""获取当前QPS"""
result = self.query_prometheus('yolo12_qps')
if result['status'] == 'success' and result['data']['result']:
return float(result['data']['result'][0]['value'][1])
return 0
def get_p95_latency(self):
"""获取P95延迟"""
result = self.query_prometheus(
'histogram_quantile(0.95, sum(rate(yolo12_inference_duration_seconds_bucket[5m])) by (le))'
)
if result['status'] == 'success' and result['data']['result']:
return float(result['data']['result'][0]['value'][1])
return 0
def should_scale_up(self):
"""判断是否需要扩容"""
gpu_util = self.get_gpu_utilization()
qps = self.get_qps()
latency = self.get_p95_latency()
# 扩容条件（满足任一即可）：
# 1. GPU利用率持续高于阈值
# 2. QPS很高但延迟也在增加
# 3. P95延迟超过阈值（如100ms）
if gpu_util > self.scale_up_threshold:
return True
if qps > 100 and latency > 0.1: # QPS>100且延迟>100ms
return True
if latency > 0.15: # P95延迟>150ms
return True
return False
def should_scale_down(self):
"""判断是否需要缩容"""
gpu_util = self.get_gpu_utilization()
qps = self.get_qps()
# 缩容条件：
# 1. GPU利用率持续低于阈值
# 2. QPS很低
if gpu_util < self.scale_down_threshold and qps < 50:
return True
return False
def scale_up(self):
"""扩容操作"""
if self.current_instances < self.max_instances:
self.current_instances += 1
# 这里调用你的部署平台API，启动新的YOLO12实例
print(f"扩容: 当前实例数 {self.current_instances}")
return True
return False
def scale_down(self):
"""缩容操作"""
if self.current_instances > self.min_instances:
self.current_instances -= 1
# 这里调用你的部署平台API，停止一个YOLO12实例
print(f"缩容: 当前实例数 {self.current_instances}")
return True
return False
def run(self):
"""主循环"""
while True:
try:
if self.should_scale_up():
self.scale_up()
elif self.should_scale_down():
self.scale_down()
# 每分钟检查一次
time.sleep(60)
except Exception as e:
print(f"自动扩缩容错误: {e}")
time.sleep(60)
# 启动自动扩缩容
if __name__ == "__main__":
scaler = YOLO12AutoScaler()
scaler.run()
7. 高级监控技巧
7.1 设置告警规则
在Prometheus中配置告警规则，当出现异常时及时通知：
# alerts.yml
groups:
- name: yolo12_alerts
rules:
- alert: HighGPUUtilization
expr: avg_over_time(yolo12_gpu_utilization_percent[5m]) > 90
for: 5m
labels:
severity: warning
annotations:
summary: "GPU利用率过高"
description: "GPU利用率持续5分钟超过90%，当前值 {{ $value }}%"
- alert: HighGPUMemoryUsage
expr: yolo12_gpu_memory_used_mb / yolo12_gpu_memory_total_mb * 100 > 85
for: 2m
labels:
severity: warning
annotations:
summary: "GPU显存使用过高"
description: "GPU显存使用率超过85%，当前值 {{ $value }}%"
- alert: LowQPS
expr: yolo12_qps < 10
for: 10m
labels:
severity: info
annotations:
summary: "QPS过低"
description: "QPS持续10分钟低于10，当前值 {{ $value }}"
- alert: HighInferenceLatency
expr: histogram_quantile(0.95, rate(yolo12_inference_duration_seconds_bucket[5m])) > 0.2
for: 5m
labels:
severity: warning
annotations:
summary: "推理延迟过高"
description: "P95推理延迟超过200ms，当前值 {{ $value }}s"
- alert: HighErrorRate
expr: sum(rate(yolo12_requests_total{status=~"5.."}[5m])) / sum(rate(yolo12_requests_total[5m])) * 100 > 5
for: 2m
labels:
severity: critical
annotations:
summary: "错误率过高"
description: "5xx错误率超过5%，当前值 {{ $value }}%"
7.2 监控数据持久化与备份
为了防止监控数据丢失，配置持久化存储和定期备份：
# 备份Prometheus数据
#!/bin/bash
# backup_prometheus.sh
BACKUP_DIR="/backup/prometheus"
DATE=$(date +%Y%m%d_%H%M%S)
# 创建备份目录
mkdir -p $BACKUP_DIR
# 停止Prometheus（短暂停止）
docker-compose stop prometheus
# 备份数据
tar -czf $BACKUP_DIR/prometheus_$DATE.tar.gz \
-C ~/yolo12-monitoring/prometheus_data .
# 启动Prometheus
docker-compose start prometheus
# 删除7天前的备份
find $BACKUP_DIR -name "prometheus_*.tar.gz" -mtime +7 -delete
echo "备份完成: prometheus_$DATE.tar.gz"
7.3 性能基准测试与对比
使用监控数据建立性能基准，方便后续对比：
# benchmark.py
import time
import statistics
import requests
from concurrent.futures import ThreadPoolExecutor
class YOLO12Benchmark:
def __init__(self, api_url, model_sizes=['yolov12n.pt', 'yolov12s.pt', 'yolov12m.pt']):
self.api_url = api_url
self.model_sizes = model_sizes
self.results = {}
def test_single_request(self, image_path, model_size):
"""测试单次请求"""
with open(image_path, 'rb') as f:
files = {'file': f}
start_time = time.time()
response = requests.post(
f"{self.api_url}/predict",
files=files,
params={'model': model_size}
)
end_time = time.time()
latency = end_time - start_time
success = response.status_code == 200
return {
'latency': latency,
'success': success,
'model_size': model_size
}
def test_concurrent(self, image_path, model_size, concurrent_users=10, duration=30):
"""测试并发性能"""
latencies = []
successes = 0
total_requests = 0
def worker():
nonlocal successes, total_requests
start_time = time.time()
while time.time() - start_time < duration:
result = self.test_single_request(image_path, model_size)
latencies.append(result['latency'])
if result['success']:
successes += 1
total_requests += 1
with ThreadPoolExecutor(max_workers=concurrent_users) as executor:
futures = [executor.submit(worker) for _ in range(concurrent_users)]
for future in futures:
future.result()
qps = total_requests / duration
return {
'model_size': model_size,
'concurrent_users': concurrent_users,
'duration': duration,
'total_requests': total_requests,
'success_rate': successes / total_requests * 100,
'qps': qps,
'avg_latency': statistics.mean(latencies) if latencies else 0,
'p95_latency': statistics.quantiles(latencies, n=20)[18] if len(latencies) >= 20 else 0,
'max_latency': max(latencies) if latencies else 0,
'min_latency': min(latencies) if latencies else 0
}
def run_benchmark(self, image_path, concurrent_users_list=[1, 5, 10, 20]):
"""运行完整的基准测试"""
for model_size in self.model_sizes:
print(f"\n测试模型: {model_size}")
print("=" * 50)
model_results = []
for concurrent_users in concurrent_users_list:
print(f"并发用户数: {concurrent_users}")
result = self.test_concurrent(
image_path, model_size,
concurrent_users, duration=30
)
model_results.append(result)
print(f" QPS: {result['qps']:.2f}")
print(f" 平均延迟: {result['avg_latency']*1000:.2f}ms")
print(f" P95延迟: {result['p95_latency']*1000:.2f}ms")
print(f" 成功率: {result['success_rate']:.2f}%")
self.results[model_size] = model_results
return self.results
def generate_report(self):
"""生成基准测试报告"""
report = "# YOLO12性能基准测试报告\n\n"
for model_size, results in self.results.items():
report += f"## 模型: {model_size}\n\n"
report += "| 并发用户 | QPS | 平均延迟(ms) | P95延迟(ms) | 成功率 |\n"
report += "|---------|-----|-------------|------------|--------|\n"
for result in results:
report += f"| {result['concurrent_users']} | {result['qps']:.2f} | {result['avg_latency']*1000:.2f} | {result['p95_latency']*1000:.2f} | {result['success_rate']:.2f}% |\n"
report += "\n"
# 保存报告
with open('benchmark_report.md', 'w') as f:
f.write(report)
print("基准测试报告已保存到 benchmark_report.md")
return report
# 使用示例
if __name__ == "__main__":
benchmark = YOLO12Benchmark(
api_url="http://localhost:8000",
model_sizes=['yolov12n.pt', 'yolov12s.pt', 'yolov12m.pt']
)
results = benchmark.run_benchmark(
image_path="test_image.jpg",
concurrent_users_list=[1, 5, 10, 20, 50]
)
benchmark.generate_report()
8. 总结
通过本文介绍的Prometheus+Grafana监控体系，你现在可以：
- 实时掌握YOLO12运行状态：随时查看GPU利用率、显存占用、QPS、延迟等关键指标
- 快速定位性能问题：通过历史数据对比，发现性能瓶颈的根本原因
- 基于数据做优化决策：根据监控数据调整模型参数、优化代码、扩容资源
- 设置智能告警：在问题发生前收到预警，避免服务中断
- 建立性能基准：为容量规划和性能优化提供数据支撑
这套监控方案不仅适用于YOLO12，也可以轻松适配其他AI模型服务。关键是要根据业务特点，监控那些真正影响用户体验和系统稳定性的指标。
监控不是目的，而是手段。真正的价值在于通过监控数据驱动优化决策，让YOLO12服务更稳定、更高效、更可靠。现在就开始搭建你的监控体系吧，让数据告诉你服务的真实状态！
获取更多AI镜像
想探索更多AI镜像和应用场景？访问 CSDN星图镜像广场，提供丰富的预置镜像，覆盖大模型推理、图像生成、视频生成、模型微调等多个领域，支持一键部署。
汇聚全球AI编程工具，助力开发者即刻编程。
更多推荐
7
0- 0
已为社区贡献50条内容
所有评论(0)