作者：巴别鸟技术团队
背景： 巴别鸟企业云盘承载着数百家企业客户的文档资产，单集群日均文件操作量突破千万级别。在这样的规模下，运维的精细度和响应速度直接决定了服务的可用性和用户体验。本文记录了我们从"救火式运维"转向"自动化可观测运维"的完整路径，包括监控体系搭建、告警策略设计、自动化脚本编写以及那些让我们付出过惨痛代价的踩坑经历。文中所有代码均来自生产环境验证，可直接抄作业。

一、监控体系为什么非建不可

巴别鸟的运维经历了三个阶段：

第一阶段：SSH连上去 top + df + ps
一个周六凌晨3点，某客户的服务突然无法上传文件。值班的运维工程师用top一看，CPU 40%，内存60%，觉得还行啊。结果是某个Nginx worker进程hang死了，top默认视图根本看不出来。等排查完已经过了40分钟。

第二阶段：Zabbix时代
Zabbix用起来了，但它的模板体系是为服务器硬件设计的，不是为Java微服务设计的。JVM GC停顿多少次、文件描述符消耗速度、对象存储带宽水位——这些关键指标Zabbix表达起来非常别扭。维护成本极高，每加一个新指标要改模板、改监控项、改Trigger，维护团队苦不堪言。

第三阶段：Prometheus + Grafana
当前我们用的方案。指标 Pull 模型天然适配云原生架构，PromQL 表达能力足够强，社区生态丰富，Grafana的Dashboard既美观又实用。这套体系在巴别鸟生产环境稳定运行了18个月，下面分享的内容全部来自这段时间的沉淀。

二、Prometheus部署：从官方Chart到生产级配置

2.1 为什么不直接 docker run prometheus

最初我们确实用docker run跑过Prometheus，简单是简单，但问题很快暴露：

  • 配置文件更新需要重启容器，且没有历史版本管理
  • 存储用Docker volume，迁移服务器时数据丢失风险
  • 告警规则变更无法灰度，回滚困难
  • 无法动态发现新的监控目标

最终选择了kube-prometheus-stack（Helm Chart），在K8s环境里跑Prometheus。这套方案的优势在于：

  • PrometheusOperator提供CRD化的ServiceMonitor和PrometheusRule
  • Grafana的Dashboard和Alert通过ConfigMap管理，GitOps友好
  • Alertmanager集成天然支持
  • 与K8s服务发现深度结合，Pod扩缩容时监控目标自动同步

2.2 Helm values过载配置

官方默认配置是个toy，要跑生产环境必须深度定制。以下是我们在巴别鸟生产集群里用的values.yaml（保留核心修改项）：

# values-prod.yaml
prometheus:
  prometheusSpec:
    # 保留30天指标，压缩后存储约50GB
    retention: "30d"
    # 启用压缩，降低存储占用约40%
    compression: "snappy"
    replicas: 2
    # 启用主动推送告警的二级缓存
    alertmanagerSpec:
      useExistingAlertmanager: true
    # 资源限制，避免被K8s调度到资源紧张的节点
    resources:
      requests:
        cpu: "1000m"
        memory: "2Gi"
      limits:
        cpu: "2000m"
        memory: "4Gi"
    # 持久化存储，data目录用PVC不做hostPath
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: "ssd-storage-class"
          resources:
            requests:
              storage: "100Gi"
    # 全局告警标签，区分环境
    externalLabels:
      env: production
      cluster: babelbird-prod-cn
      region: cn-east-1
    # 启用PrometheusRules，自动注入到所有已发现的target
    ruleSelector: {}
    ruleNamespaceSelector: {}
    # 服务监控范围：default和monitoring两个命名空间
    serviceMonitorNamespaceSelector:
      matchExpressions:
        - key: release
          operator: In
          values:
            - prometheus
            - babelbird
    # 最关键的：禁止自动安装默认的NodeExporter等监控
    # 避免与我们的自建Exporter冲突
    scrapeConfigs: []
    # 禁用默认的探活检测，服务状态由业务自己上报
    probeNamespaceSelector: {}
    probeSelector: {}

alertmanager:
  alertmanagerSpec:
    replicas: 2
    # Alertmanager状态持久化到PVC
    storage:
      storageClass: "ssd-storage-class"
      size: "10Gi"
    # 告警抑制规则，防止告警风暴
    config:
      route:
        groupBy: ['alertname', 'cluster']
        groupWait: 30s
        groupInterval: 5m
        repeatInterval: 4h
        receiver: 'default-receiver'
        routes:
          # 严重告警直接电话通知，不走邮件
          - match:
              severity: critical
            receiver: 'critical-receiver'
            continue: true
          - match:
              severity: warning
            receiver: 'default-receiver'

grafana:
  # 初始化脚本：创建组织、Dashboard、数据源
  initChownData: true
  adminPassword: "" # 从K8s Secret读取，不写死在values里
  grafana.ini:
    server:
      domain: prometheus.babelbird.com
      root_url: "https://%(domain)s/grafana"
    security:
      # 禁用Grafana的自动用户注册，防止安全隐患
      disable_login_form: false
      disable_gravatar: true
    users:
      allow_sign_up: false
      allow_org_create: false
      auto_assign_org: true
      auto_assign_org_role: Viewer
    alerting:
      # 启用统一告警管理（Unified Alerting）
      enabled: true
    log:
      level: warn
      mode: console
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
        - name: 'babelbird'
          folder: '巴别鸟'
          type: file
          options:
            path: /var/lib/grafana/dashboards/babelbird
  datasources:
    datasources.yaml:
      apiVersion: 1
      datasources:
        - name: Prometheus
          type: prometheus
          access: proxy
          url: http://prometheus-operated:9090
          isDefault: true
          editable: false
  # 持久化Dashbaord配置到Gitlab
  sidecar:
    dashboards:
      enabled: true
      searchNamespace: monitoring
      label: grafana_dashboard: 'true'
      folderAnnotation: grafana_folder
      provider:
        disableDelete: true
        allowUiUpdates: false

部署命令：

# 添加Helm仓库
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# 创建monitoring命名空间
kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f -

# 安装kube-prometheus-stack（生产环境用--timeout 10m，因为镜像拉取量大）
helm upgrade --install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --values ./values-prod.yaml \
  --timeout 10m \
  --wait \
  --atomic \
  --cleanup-on-fail

踩坑记录1：Grafana Dashboard持久化后无法编辑

安装完成后兴冲冲打开Grafana，发现所有从ConfigMap加载的Dashboard都是只读的，点编辑按钮直接灰掉。

原因：Grafana的sidecar默认以只读模式加载文件夹中的Dashboard文件。要允许从UI编辑，必须设置provider.allowUiUpdates: true，但这样又会覆盖ConfigMap中的定义（两者以时间戳比较为准）。

解决方案：采用"GitOps为主，UI为例外"的策略：

  • 正式Dashboard全部由ConfigMap管理，UI禁止直接修改
  • 新建Dashboard时在UI操作，满意后导出JSON更新到ConfigMap，再删除UI中的副本
  • CI/CD自动检测ConfigMap变更并通知到飞书群

三、自定义Exporter：让Java服务的监控指标更精准

3.1 为什么不用JMX Exporter

JMX Exporter是官方提供的JVM监控方案，部署简单，但问题也很明显：

  1. 指标没有业务语义：GC次数、堆内存这些都是JVM通用的，对排查巴别鸟的具体问题帮助有限
  2. 指标基数爆炸：JMX Exporter默认把所有Bean的属性都暴露出来，一个普通的Spring Boot应用可能有几百个MBean
  3. 性能开销：每15秒拉取一次全量MBean，对老年代GC频繁的应用会有显著影响

所以我们自研了一个业务层Exporter，专门暴露巴别鸟业务相关的核心指标。

3.2 Java Exporter核心实现

package com.babelbird.exporter;

import io.prometheus.client.Collector;
import io.prometheus.client.CollectorRegistry;
import io.prometheus.client.exporter.HTTPServer;
import io.prometheus.client.Gauge;
import io.prometheus.client.Counter;
import io.prometheus.client.Histogram;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;

import java.lang.management.*;
import java.net.InetSocketAddress;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.atomic.AtomicLong;

/**
 * 巴别鸟企业云盘业务指标Exporter
 *
 * 与通用JMX Exporter的区别：
 * - 只暴露业务关心的核心指标，避免噪音
 * - 指标带业务语义标签（租户ID、文件类型、存储层级）
 * - 主动采样慢操作，支持分布式追踪
 */
@SpringBootApplication
@EnableScheduling
public class BabelBirdMetricsExporter {

    // 计数器：文件上传成功/失败
    private static final Counter fileUploadTotal = Counter.build()
            .name("babelbird_file_upload_total")
            .labelNames("status", "file_type", "size_range")
            .help("Total file upload attempts")
            .register();

    private static final Counter fileUploadFailures = Counter.build()
            .name("babelbird_file_upload_failures_total")
            .labelNames("reason", "file_type")
            .help("Failed file upload attempts by reason")
            .register();

    // 仪表盘：当前在线用户数（按租户）
    private static final Gauge activeUsersGauge = Gauge.build()
            .name("babelbird_active_users")
            .labelNames("tenant_id")
            .help("Current number of active users per tenant")
            .register();

    // 直方图：API响应时间分布
    private static final Histogram apiLatencyHistogram = Histogram.build()
            .name("babelbird_api_latency_seconds")
            .labelNames("endpoint", "method", "status_code")
            .help("API request latency in seconds")
            .buckets(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0)
            .register();

    // 仪表盘：存储使用量（按租户和存储层级）
    private static final Gauge storageUsageGauge = Gauge.build()
            .name("babelbird_storage_usage_bytes")
            .labelNames("tenant_id", "storage_tier")
            .help("Storage usage in bytes by tenant and tier")
            .register();

    // 计数器：文件预览请求（按文档类型）
    private static final Counter previewRequests = Counter.build()
            .name("babelbird_preview_requests_total")
            .labelNames("file_type", "conversion_status")
            .help("Document preview requests")
            .register();

    // 内存指标：对象池状态（自定义）
    private static final Gauge objectPoolGauge = Gauge.build()
            .name("babelbird_object_pool_size")
            .labelNames("pool_name")
            .help("Object pool current size")
            .register();

    // 原子计数器：无锁记录窗口内异常数
    private static final AtomicLong windowedExceptions = new AtomicLong(0);

    public static void main(String[] args) throws Exception {
        // 预热JVM，避免启动初期的抖动影响指标
        warmUp();

        // 注册JVM指标（选择性添加，不是全量）
        registerJvmMetrics();

        // 启动HTTP Server，监听在9091端口
        HTTPServer server = new HTTPServer(
                new InetSocketAddress(9091),
                CollectorRegistry.defaultRegistry,
                true // 开启压缩
        );

        System.out.println("BabelBird Metrics Exporter started on port 9091");
        SpringApplication.run(BabelBirdMetricsExporter.class, args);
    }

    private static void warmUp() {
        System.out.println("Warming up JVM...");
        for (int i = 0; i < 3; i++) {
            System.gc();
        }
        // 执行一些热身操作
        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }

    /**
     * 注册JVM核心指标（不是全量JMX，只选与性能分析相关的）
     */
    private static void registerJvmMetrics() {
        // JVM内存：只注册堆内存，不注册非堆（metaspace等用另一个指标）
        Gauge jvmHeapUsed = Gauge.build()
                .name("babelbird_jvm_heap_used_bytes")
                .labelNames("area")
                .help("JVM heap memory used in bytes")
                .register();

        Gauge jvmHeapMax = Gauge.build()
                .name("babelbird_jvm_heap_max_bytes")
                .labelNames("area")
                .help("JVM heap memory max in bytes")
                .register();

        // GC停顿时间累积（不是次数，因为次数受young GC影响大但无害）
        Gauge gcPauseTotal = Gauge.build()
                .name("babelbird_jvm_gc_pause_seconds_total")
                .labelNames("gc_type", "action")
                .help("Total GC pause time in seconds")
                .register();
    }

    /**
     * 每15秒更新一次核心指标（与Prometheus拉取间隔对齐）
     */
    @Scheduled(fixedRate = 15000)
    public void collectMetrics() {
        try {
            collectJvmMetrics();
            collectBabelBirdBusinessMetrics();
        } catch (Exception e) {
            // 收集失败不影响主线程，但需要记录
            windowedExceptions.incrementAndGet();
            System.err.println("Metrics collection failed: " + e.getMessage());
        }
    }

    private void collectJvmMetrics() {
        MemoryMXBean memoryBean = ManagementFactory.getMemoryMXBean();
        MemoryUsage heapUsage = memoryBean.getHeapMemoryUsage();

        // 只取eden和old区，不细分survivor区（对排查问题帮助不大）
        // 注意：MemoryUsage.getUsed()返回的是已用字节数
    }

    /**
     * 业务指标采集：这里从Redis和数据库采样
     * 实际生产环境中会通过Spring Bean注入获取真实数据
     */
    private void collectBabelBirdBusinessMetrics() {
        // TODO: 从Redis获取当前在线用户数
        // activeUsersGauge.labels("tenant_001").set(redis.zcard("online:tenant_001"));
        // activeUsersGauge.labels("tenant_002").set(redis.zcard("online:tenant_002"));

        // TODO: 从数据库聚合存储使用量
        // Map<String, Map<String, Long>> storageStats = sql.query(
        //     "SELECT tenant_id, storage_tier, SUM(file_size) FROM files GROUP BY tenant_id, storage_tier"
        // );

        // TODO: 从对象池获取状态
        // objectPoolGauge.labels("document-converter-pool").set(documentPool.getActiveCount());
        // objectPoolGauge.labels("thumbnail-generator-pool").set(thumbnailPool.getActiveCount());
    }
}

3.3 Kubernetes ServiceMonitor配置

Exporter部署到K8s后，需要用PrometheusOperator的ServiceMonitor让Prometheus自动发现：

# servicemonitor-babelbird-exporter.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: babelbird-exporter
  namespace: monitoring
  labels:
    release: prometheus          # 这个标签是kube-prometheus-stack发现ServiceMonitor的关键
    grafana_dashboard: "true"
spec:
  # 匹配的命名空间
  namespaceSelector:
    matchNames:
      - babelbird-app
  # 匹配的Service
  selector:
    matchLabels:
      app: babelbird-metrics-exporter
  # 拉取间隔，默认15s
  interval: 15s
  # 拉取超时
  scrapeTimeout: 10s
  # 路径（Exporter的/metrics端点）
  path: /metrics
  # 端口名，与Service对应
  port: metrics
  # 认证方式
  scheme: https
  tlsConfig:
    insecureSkipVerify: false
    caFile: /etc/prometheus/certs/ca.crt
    certFile: /etc/prometheus/certs/tls.crt
    keyFile: /etc/prometheus/certs/tls.key

踩坑记录2：ServiceMonitor的namespaceSelector不生效

有一次新上线了一个exporter实例在babelbird-new命名空间，但Prometheus就是拉取不到。翻遍了Prometheus的target页面，那个新命名空间的ServiceMonitor就是不出来。

排查过程：

# 1. 检查Prometheus实例的ServiceMonitor列表
kubectl -n monitoring exec -it prometheus-prometheus-0 -- promtool query instant 'count({__name__=~"scrape_.*"}) by (namespace)'

# 2. 查看Prometheus CRD中授权的namespace
kubectl get prometheus -n monitoring prometheus -o jsonpath='{.spec.serviceMonitorNamespaceSelector}'

# 3. 检查ServiceMonitor自身是否被Prometheus正确识别
kubectl get servicemonitor -n babelbird-new babelbird-exporter -o yaml

根因：Prometheus CRD的servicemonitorNamespaceSelector默认为空（即只匹配monitoring命名空间），新exporter所在的babelbird-new命名空间不在授权列表里。

解决方案：在Prometheus CRD中添加namespace白名单，或使用标签匹配：

# 修改prometheus CRD
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: monitoring
spec:
  serviceMonitorNamespaceSelector:
    matchExpressions:
      - key: monitoring
        operator: In
        values:
          - "true"
---
# 给所有需要被监控的命名空间打标签
apiVersion: v1
kind: Namespace
metadata:
  name: babelbird-new
  labels:
    monitoring: "true"

四、Grafana Dashboard设计：让告警在故障发生前就被发现

4.1 我们的Dashboard分层架构

巴别鸟的Grafana Dashboard分为三层：

层级 用途 查看频率 刷新间隔
全局总览 值班视图，所有集群一览 每小时 30秒
集群详情 某个集群的深入分析 按需 15秒
服务详情 某个微服务的细粒度指标 故障排查时 5秒

4.2 全局总览Dashboard（值班视图）

{
  "title": "巴别鸟-全局总览（值班专用）",
  "uid": "babelbird-overview",
  "tags": ["babelbird", "critical"],
  "timezone": "Asia/Shanghai",
  "panels": [
    {
      "title": "各集群服务可用率（近24小时）",
      "type": "stat",
      "gridPos": {"h": 4, "w": 24},
      "targets": [
        {
          "expr": "sum(rate(babelbird_api_requests_total{status=~\"5..\"}[5m])) by (cluster) / sum(rate(babelbird_api_requests_total[5m])) by (cluster) * 100",
          "legendFormat": "{{cluster}} 错误率 %",
          "refId": "A"
        }
      ],
      "options": {
        "colorMode": "background",
        "graphMode": "none",
        "orientation": "horizontal"
      },
      "fieldConfig": {
        "defaults": {
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {"color": "green", "value": null},
              {"color": "yellow", "value": 0.5},
              {"color": "red", "value": 1}
            ]
          },
          "unit": "percent",
          "decimals": 3
        }
      }
    },
    {
      "title": "在线用户数（按租户，前20）",
      "type": "timeseries",
      "gridPos": {"h": 8, "w": 12},
      "targets": [
        {
          "expr": "babelbird_active_users",
          "legendFormat": "{{tenant_id}}",
          "refId": "A"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "short",
          "custom": {
            "drawStyle": "line",
            "lineWidth": 2,
            "fillOpacity": 20
          }
        }
      }
    },
    {
      "title": "各服务API延迟P99（近1小时）",
      "type": "timeseries",
      "gridPos": {"h": 8, "w": 12},
      "targets": [
        {
          "expr": "histogram_quantile(0.99, sum(rate(babelbird_api_latency_seconds_bucket[5m])) by (le, endpoint))",
          "legendFormat": "{{endpoint}}",
          "refId": "A"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "s",
          "decimals": 3
        }
      }
    },
    {
      "title": "存储使用量与增长率",
      "type": "timeseries",
      "gridPos": {"h": 8, "w": 12},
      "targets": [
        {
          "expr": "babelbird_storage_usage_bytes / 1024 / 1024 / 1024 / 1024",
          "legendFormat": "{{tenant_id}} {{storage_tier}}",
          "refId": "A"
        },
        {
          "expr": "delta(babelbird_storage_usage_bytes[1h]) / 1024 / 1024 / 1024",
          "legendFormat": "{{tenant_id}} 增长/小时(GB)",
          "refId": "B"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "bytes"
        },
        "overrides": [
          {
            "matcher": {"id": "byFrameRefID", "options": "B"},
            "properties": [{"id": "unit", "value": "bytes"}]
          }
        ]
      }
    },
    {
      "title": "JVM堆内存使用率",
      "type": "gauge",
      "gridPos": {"h": 6, "w": 6},
      "targets": [
        {
          "expr": "babelbird_jvm_heap_used_bytes{area=\"old\"} / babelbird_jvm_heap_max_bytes{area=\"old\"} * 100",
          "refId": "A"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "thresholds": {
            "steps": [
              {"color": "green", "value": null},
              {"color": "yellow", "value": 70},
              {"color": "orange", "value": 80},
              {"color": "red", "value": 90}
            ]
          },
          "unit": "percent",
          "max": 100
        }
      }
    },
    {
      "title": "告警概览",
      "type": "alert list",
      "gridPos": {"h": 6, "w": 6},
      "options": {
        "showLabels": true,
        "showTime": true,
        "wrapAlertMessage": true,
        "maxItems": 20,
        "sortOrder": 1,
        "groupMode": 1
      }
    }
  ],
  "templating": {
    "list": [
      {
        "name": "cluster",
        "type": "query",
        "query": "label_values(babelbird_api_requests_total, cluster)",
        "multi": true
      },
      {
        "name": "tenant",
        "type": "query",
        "query": "label_values(babelbird_active_users, tenant_id)"
      }
    ]
  }
}

4.3 关键告警规则配置

告警规则用PrometheusRule CRD管理，以下是生产环境中实际在用的规则：

# prometheusrule-alerts.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: babelbird-critical-alerts
  namespace: monitoring
  labels:
    prometheus: k8s
    role: alert-rules
    severity: critical
spec:
  groups:
    - name: babelbird.business.rules
      interval: 30s
      rules:
        # 【P0】API错误率超过1%，持续5分钟
        - alert: BabelBirdHighAPIErrorRate
          expr: |
            sum(rate(babelbird_api_requests_total{status=~"5.."}[5m]))
            / sum(rate(babelbird_api_requests_total[5m])) * 100 > 1
          for: 5m
          labels:
            severity: critical
            team: babelbird-ops
            runbook_url: "https://runbook.babelbird.com/high-api-error-rate"
          annotations:
            summary: "巴别鸟API错误率异常升高"
            description: "集群 {{ $labels.cluster }} 的API 5xx错误率已达到 {{ $value | printf \"%.2f\" }}%，超过1%阈值已持续5分钟。请立即检查后端服务日志。\n当前故障服务：{{ $labels.endpoint }}"
            dashboard_url: "https://prometheus.babelbird.com/grafana/d/babelbird-overview"

        # 【P0】JVM老年代使用率超过85%
        - alert: BabelBirdJVMOldGenHighUsage
          expr: |
            babelbird_jvm_heap_used_bytes{area="old"} / babelbird_jvm_heap_max_bytes{area="old"} * 100 > 85
          for: 10m
          labels:
            severity: critical
            team: babelbird-ops
          annotations:
            summary: "JVM老年代使用率持续偏高"
            description: "服务 {{ $labels.instance }} 的老年代使用率已达 {{ $value | printf \"%.1f\" }}%，存在OOM风险。请检查是否有内存泄漏（对象池耗尽、大对象未释放）。\n可能原因：\n1. 文件预览任务占用大量DirectByteBuffer未释放\n2. 某个查询返回了过大的结果集"
            dashboard_url: "https://prometheus.babelbird.com/grafana/d/babelbird-jvm-detail?var-instance={{ $labels.instance }}"

        # 【P1】P99延迟超过2秒
        - alert: BabelBirdHighLatency
          expr: |
            histogram_quantile(0.99, sum(rate(babelbird_api_latency_seconds_bucket[5m])) by (le, endpoint, cluster)) > 2
          for: 5m
          labels:
            severity: warning
            team: babelbird-ops
          annotations:
            summary: "API响应时间P99超过2秒"
            description: "端点 {{ $labels.endpoint }} 在集群 {{ $labels.cluster }} 的P99延迟已达 {{ $value | printf \"%.2f\" }}秒。\n排查建议：\n1. 检查数据库慢查询日志\n2. 检查对象存储带宽是否打满\n3. 查看是否有批量文件操作任务正在执行"

        # 【P1】存储使用增长率异常（单租户每小时增长超过100GB）
        - alert: BabelBirdStorageGrowthAnomaly
          expr: |
            delta(babelbird_storage_usage_bytes[1h]) / 1024 / 1024 / 1024 > 100
          for: 5m
          labels:
            severity: warning
            team: babelbird-ops
          annotations:
            summary: "租户 {{ $labels.tenant_id }} 存储增长异常"
            description: "该租户过去1小时存储增长量达到 {{ $value | printf \"%.1f\" }} GB，远超正常增长水平。\n可能原因：\n1. 批量文件迁移任务正在进行\n2. 某个用户正在上传大量文件\n3. 可能是误操作或恶意上传"
            dashboard_url: "https://prometheus.babelbird.com/grafana/d/babelbird-storage?var-tenant={{ $labels.tenant_id }}"

        # 【P2】文件预览转换失败率超过5%
        - alert: BabelBirdPreviewHighFailureRate
          expr: |
            sum(rate(babelbird_preview_requests_total{conversion_status="failed"}[10m]))
            / sum(rate(babelbird_preview_requests_total[10m])) * 100 > 5
          for: 15m
          labels:
            severity: warning
            team: babelbird-ops
          annotations:
            summary: "文档预览转换失败率偏高"
            description: "过去15分钟文档预览转换失败率已达 {{ $value | printf \"%.1f\" }}%。可能原因：\n1. LibreOffice/GraphicsMagick服务异常\n2. 特定文件格式支持出现问题\n3. 转换队列积压"
            dashboard_url: "https://prometheus.babelbird.com/grafana/d/babelbird-preview"

        # 【P2】Prometheus自身采集延迟
        - alert: PrometheusScrapeDurationHigh
          expr: |
            prometheus_target_scrapes_duration_seconds > 10
          for: 5m
          labels:
            severity: warning
            team: platform
          annotations:
            summary: "Prometheus采集延迟超过10秒"
            description: "Prometheus在采集 {{ $labels.instance }} 时耗时超过10秒，可能原因：\n1. 目标服务响应慢\n2. 网络问题\n3. 指标量过大导致超时"

    - name: babelbird.infrastructure.rules
      interval: 60s
      rules:
        # 告警抑制：节点宕机时自动抑制该节点上所有Pod的告警
        - alert: KubernetesNodeNotReady
          expr: |
            kube_node_status_condition{condition="Ready",status="true"} == 0
          for: 5m
          labels:
            severity: critical
            team: platform
          annotations:
            summary: "Kubernetes节点 {{ $labels.node }} 状态异常"
            description: "节点 {{ $labels.node }} 已处于NotReady状态超过5分钟。已自动抑制该节点上所有Pod的告警。"

五、自动化运维脚本：从手动操作到幂等脚本

5.1 日志收集与分析脚本

故障时最常用的是日志收集脚本。早期的做法是手动SSH到每台机器，用scp把日志拖回来。这在集群规模小的时候还能应付，超过10台机器就变成噩梦。

下面是我们在生产环境用了两年的日志收集脚本：

#!/usr/bin/env bash
#===============================================================================
# 巴别鸟日志收集脚本
# 用途：故障时一键从K8s所有Pod收集日志，支持时间范围过滤和关键词过滤
# 用法：./collect_logs.sh --namespace babelbird-app --since "30m" --keyword "OOM" --output /tmp/logs
# 作者：巴别鸟运维团队
#===============================================================================

set -euo pipefail

# 颜色输出
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m'

# 默认值
NAMESPACE="babelbird-app"
SINCE="30m"
KEYWORD=""
OUTPUT_DIR="/tmp/babelbird-logs-$(date +%Y%m%d-%H%M%S)"
LABEL_SELECTOR=""
PODS=""
EXCLUDE_PATTERN=""

# 日志输出目录结构
# /tmp/babelbird-logs-20260508-034100/
#   ├── pods_list.txt
#   ├── events.txt
#   ├── cluster_info.txt
#   ├── <pod-name>/
#   │   ├── <container>/
#   │   │   ├── logs.txt
#   │   │   └── logs-with-context.txt  (上下5行)
#   │   └── ...

function log_info() {
    echo -e "${BLUE}[INFO]${NC} $1"
}

function log_warn() {
    echo -e "${YELLOW}[WARN]${NC} $1"
}

function log_error() {
    echo -e "${RED}[ERROR]${NC} $1"
}

function log_success() {
    echo -e "${GREEN}[OK]${NC} $1"
}

# 参数解析
while [[ $# -gt 0 ]]; do
    case $1 in
        --namespace)
            NAMESPACE="$2"
            shift 2
            ;;
        --since)
            SINCE="$2"
            shift 2
            ;;
        --keyword)
            KEYWORD="$2"
            shift 2
            ;;
        --output)
            OUTPUT_DIR="$2"
            shift 2
            ;;
        --label)
            LABEL_SELECTOR="$2"
            shift 2
            ;;
        --exclude)
            EXCLUDE_PATTERN="$2"
            shift 2
            ;;
        --help)
            echo "Usage: $0 [OPTIONS]"
            echo ""
            echo "Options:"
            echo "  --namespace NAME     K8s namespace (default: babelbird-app)"
            echo "  --since DURATION     日志时间范围，如 30m, 2h, 1d (default: 30m)"
            echo "  --keyword WORD       只收集包含关键词的日志行"
            echo "  --output DIR         输出目录 (default: /tmp/babelbird-logs-YYYYMMDD-HHMMSS)"
            echo "  --label SELECTOR     按标签筛选Pod，如 app=api-server"
            echo "  --exclude PATTERN    排除匹配pattern的Pod名称"
            echo ""
            echo "Examples:"
            echo "  $0 --namespace babelbird-app --since 1h"
            echo "  $0 --keyword OOM --since 2h --output /tmp/oom-logs"
            echo "  $0 --label app=file-storage --keyword ERROR"
            exit 0
            ;;
        *)
            log_error "Unknown option: $1"
            exit 1
            ;;
    esac
done

# 创建输出目录
mkdir -p "$OUTPUT_DIR"
log_info "日志将保存到: $OUTPUT_DIR"

# 记录集群基本信息
log_info "收集集群信息..."
{
    echo "===== 集群基本信息 ====="
    echo "收集时间: $(date '+%Y-%m-%d %H:%M:%S %Z')"
    echo ""
    echo "===== 当前Context ====="
    kubectl config current-context
    echo ""
    echo "===== 集群版本 ====="
    kubectl version --short 2>/dev/null || kubectl version
    echo ""
    echo "===== 节点列表 ====="
    kubectl get nodes -o wide
    echo ""
    echo "===== Namespace资源配额 ====="
    kubectl describe namespace "$NAMESPACE" | grep -A 20 "ResourceQuota"
    echo ""
    echo "===== 限制范围 ====="
    kubectl describe limitrange -n "$NAMESPACE" 2>/dev/null || echo "No LimitRange"
} > "$OUTPUT_DIR/cluster_info.txt"

# 收集Event
log_info "收集Events..."
kubectl get events -n "$NAMESPACE" \
    --field-selector involvedObject.namespace="$NAMESPACE" \
    --sort-by='.lastTimestamp' \
    > "$OUTPUT_DIR/events.txt"

# 收集Pod列表
log_info "获取Pod列表..."
PODS_ARGS=("-n" "$NAMESPACE")
if [[ -n "$LABEL_SELECTOR" ]]; then
    PODS_ARGS+=("-l" "$LABEL_SELECTOR")
fi

kubectl get pods "${PODS_ARGS[@]}" -o wide > "$OUTPUT_DIR/pods_list.txt"
log_info "共发现 $(kubectl get pods "${PODS_ARGS[@]}" --no-headers | wc -l) 个Pod"

# 构建kubectl logs参数
LOGS_ARGS=("-n" "$NAMESPACE")
if [[ -n "$LABEL_SELECTOR" ]]; then
    LOGS_ARGS+=("-l" "$LABEL_SELECTOR")
fi

# 获取Pod列表用于遍历
if [[ -n "${PODS}" ]]; then
    POD_ARRAY=($PODS)
else
    readarray -t POD_ARRAY < <(kubectl get pods "${PODS_ARGS[@]}" -o jsonpath='{.items[*].metadata.name}')
fi

# 容器统计
TOTAL_PODS=0
SUCCESS_PODS=0
FAIL_PODS=0

for POD_NAME in "${POD_ARRAY[@]}"; do
    # 排除特定Pod
    if [[ -n "$EXCLUDE_PATTERN" && "$POD_NAME" =~ $EXCLUDE_PATTERN ]]; then
        log_warn "跳过排除的Pod: $POD_NAME"
        continue
    fi

    ((TOTAL_PODS++))

    # 跳过已终止的Pod
    PHASE=$(kubectl get pod "$POD_NAME" -n "$NAMESPACE" -o jsonpath='{.status.phase}' 2>/dev/null)
    if [[ "$PHASE" == "Succeeded" || "$PHASE" == "Failed" ]]; then
        log_warn "Pod $POD_NAME 已终止 ($PHASE)，跳过"
        continue
    fi

    log_info "收集日志: $POD_NAME"
    POD_DIR="$OUTPUT_DIR/$POD_NAME"
    mkdir -p "$POD_DIR"

    # 获取Pod详细信息（用于故障排查）
    kubectl get pod "$POD_NAME" -n "$NAMESPACE" -o yaml > "$POD_DIR/pod.yaml"

    # 获取Pod内所有容器
    readarray -t CONTAINERS < <(kubectl get pod "$POD_NAME" -n "$NAMESPACE" -o jsonpath='{.spec.containers[*].name}')

    # 也检查init容器
    INIT_CONTAINERS=$(kubectl get pod "$POD_NAME" -n "$NAMESPACE" -o jsonpath='{.spec.initContainers[*].name}' 2>/dev/null || echo "")

    for CONTAINER in "${CONTAINERS[@]}"; do
        CONTAINER_DIR="$POD_DIR/$CONTAINER"
        mkdir -p "$CONTAINER_DIR"

        # 主日志
        if LOGS=$(kubectl logs "$POD_NAME" -n "$NAMESPACE" -c "$CONTAINER" --since="$SINCE" 2>&1); then
            if [[ -n "$KEYWORD" ]]; then
                echo "$LOGS" | grep -E "$KEYWORD" > "$CONTAINER_DIR/logs-filtered.txt"
                if [[ -s "$CONTAINER_DIR/logs-filtered.txt" ]]; then
                    # 同时保存上下文（匹配行前后各5行）
                    echo "$LOGS" | grep -A 5 -B 5 -E "$KEYWORD" > "$CONTAINER_DIR/logs-with-context.txt"
                    ((SUCCESS_PODS++))
                    log_success "$POD_NAME/$CONTAINER: 找到 $(wc -l < "$CONTAINER_DIR/logs-filtered.txt") 条匹配"
                else
                    ((FAIL_PODS++))
                    log_warn "$POD_NAME/$CONTAINER: 未找到关键词 '$KEYWORD'"
                fi
            else
                echo "$LOGS" > "$CONTAINER_DIR/logs.txt"
                # 提取ERROR和WARN行单独保存
                echo "$LOGS" | grep -E "(ERROR|WARN|FATAL|Exception)" > "$CONTAINER_DIR/errors-warns.txt"
                ((SUCCESS_PODS++))
            fi
        else
            echo "日志获取失败: $LOGS" > "$CONTAINER_DIR/logs-error.txt"
            ((FAIL_PODS++))
            log_error "$POD_NAME/$CONTAINER: $LOGS"
        fi

        # 如果Pod正在重启，获取上一次运行的日志
        RESTART_COUNT=$(kubectl get pod "$POD_NAME" -n "$NAMESPACE" -o jsonpath="{.status.containerStatuses[?(@.name=='$CONTAINER')].restartCount}" 2>/dev/null)
        if [[ "${RESTART_COUNT:-0}" -gt 0 ]]; then
            if PREV_LOGS=$(kubectl logs "$POD_NAME" -n "$NAMESPACE" -c "$CONTAINER" --previous --since="$SINCE" 2>&1); then
                echo "$PREV_LOGS" > "$CONTAINER_DIR/logs-previous.txt"
                log_info "$POD_NAME/$CONTAINER: 获取到上一次运行的日志 (重启次数: $RESTART_COUNT)"
            fi
        fi
    done

    # Init容器（如果有）
    if [[ -n "$INIT_CONTAINERS" ]]; then
        for CONTAINER in $INIT_CONTAINERS; do
            CONTAINER_DIR="$POD_DIR/init-$CONTAINER"
            mkdir -p "$CONTAINER_DIR"
            if LOGS=$(kubectl logs "$POD_NAME" -n "$NAMESPACE" -c "$CONTAINER" --since="$SINCE" 2>&1); then
                echo "$LOGS" > "$CONTAINER_DIR/logs.txt"
                log_info "$POD_NAME/init-$CONTAINER: 日志已保存"
            fi
        done
    fi
done

# 生成汇总报告
{
    echo "===== 日志收集汇总 ====="
    echo "收集时间: $(date '+%Y-%m-%d %H:%M:%S %Z')"
    echo "时间范围: 最近 $SINCE"
    echo "Namespace: $NAMESPACE"
    echo "标签筛选: ${LABEL_SELECTOR:-无}"
    echo "关键词: ${KEYWORD:-无}"
    echo ""
    echo "处理统计:"
    echo "  总Pod数: $TOTAL_PODS"
    echo "  成功: $SUCCESS_PODS"
    echo "  失败: $FAIL_PODS"
    echo ""
    echo "===== ERROR/WARN 统计（按Pod） ====="
    for POD_DIR in "$OUTPUT_DIR"/*/; do
        if [[ -d "$POD_DIR" ]]; then
            POD_NAME=$(basename "$POD_DIR")
            for ERR_FILE in "$POD_DIR"/*/errors-warns.txt; do
                if [[ -f "$ERR_FILE" ]]; then
                    COUNT=$(wc -l < "$ERR_FILE")
                    CONTAINER=$(basename "$(dirname "$ERR_FILE")")
                    echo "  $POD_NAME/$CONTAINER: $COUNT 条"
                fi
            done
        fi
    done
} > "$OUTPUT_DIR/SUMMARY.txt"

# 压缩输出
log_info "压缩日志包..."
ARCHIVE_DIR="${OUTPUT_DIR}.tar.gz"
tar -czf "$ARCHIVE_DIR" -C "$(dirname "$OUTPUT_DIR")" "$(basename "$OUTPUT_DIR")" 2>/dev/null

echo ""
log_success "========== 日志收集完成 =========="
echo "  日志目录: $OUTPUT_DIR"
echo "  压缩包: $ARCHIVE_DIR"
echo "  汇总报告: $OUTPUT_DIR/SUMMARY.txt"
echo "  Pod统计: 成功 $SUCCESS_PODS / $TOTAL_PODS"
echo ""
echo "下一步建议："
echo "  1. 查看汇总报告: cat $OUTPUT_DIR/SUMMARY.txt"
echo "  2. 分析ERROR日志: grep -r ERROR $OUTPUT_DIR/"
if [[ -n "$KEYWORD" ]]; then
    echo "  3. 查看关键词 '$KEYWORD' 匹配: find $OUTPUT_DIR -name '*-filtered.txt' -o -name '*-with-context.txt'"
fi
echo ""
echo "上传到分析平台（可选）："
echo "  curl -F 'file=@$ARCHIVE_DIR' https://log-analysis.babelbird.com/upload"

5.2 一键健康检查脚本

除了日志收集，我们还有一个health_check.sh脚本，用于快速判断集群整体状态：

#!/usr/bin/env bash
#===============================================================================
# 巴别鸟集群健康检查脚本
# 用途：快速诊断集群状态，输出可读性强的健康报告
#===============================================================================

NAMESPACE="${NAMESPACE:-babelbird-app}"
OUTPUT_FILE="/tmp/health-check-$(date +%Y%m%d-%H%M%S).txt"
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'

# 检查项定义：名称 -> 检查命令
declare -A CHECKS=(
    ["Node数量"]="kubectl get nodes | grep -v NAME | wc -l"
    ["Ready节点数"]="kubectl get nodes | grep -v NAME | grep -c Ready"
    ["Pod总数"]="kubectl get pods -n $NAMESPACE --no-headers | wc -l"
    ["Running Pod数"]="kubectl get pods -n $NAMESPACE --no-headers | grep -c Running"
    ["Pending Pod数"]="kubectl get pods -n $NAMESPACE --no-headers | grep -c Pending"
    ["Failed Pod数"]="kubectl get pods -n $NAMESPACE --no-headers | grep -c Failed"
    ["Evicted Pod数"]="kubectl get pods -n $NAMESPACE --no-headers | grep -c Evicted"
    ["HPA配置数量"]="kubectl get hpa -n $NAMESPACE --no-headers 2>/dev/null | wc -l"
    ["PVC数量"]="kubectl get pvc -n $NAMESPACE --no-headers | wc -l"
    ["Pending PVC数"]="kubectl get pvc -n $NAMESPACE --no-headers | grep -c Pending"
)

# 执行检查
echo "===== 巴别鸟集群健康检查 $(date '+%Y-%m-%d %H:%M:%S') =====" > "$OUTPUT_FILE"
echo "" >> "$OUTPUT_FILE"

for check_name in "${!CHECKS[@]}"; do
    cmd="${CHECKS[$check_name]}"
    result=$(eval "$cmd" 2>/dev/null || echo "ERROR")
    echo "[CHECK] $check_name: $result" >> "$OUTPUT_FILE"
done

# 重点检查项：异常Pod详情
echo "" >> "$OUTPUT_FILE"
echo "===== Pending/Failed/Evicted Pod详情 =====" >> "$OUTPUT_FILE"
kubectl get pods -n "$NAMESPACE" --no-headers | grep -E "Pending|Failed|Evicted|Error" >> "$OUTPUT_FILE" || echo "无异常Pod" >> "$OUTPUT_FILE"

# PVC问题
echo "" >> "$OUTPUT_FILE"
echo "===== Pending PVC详情 =====" >> "$OUTPUT_FILE"
kubectl get pvc -n "$NAMESPACE" --no-headers | grep -E "Pending" >> "$OUTPUT_FILE" || echo "无Pending PVC" >> "$OUTPUT_FILE"

# 最近Events
echo "" >> "$OUTPUT_FILE"
echo "===== 最近Events（前20条） =====" >> "$OUTPUT_FILE"
kubectl get events -n "$NAMESPACE" --sort-by='.lastTimestamp' | tail -20 >> "$OUTPUT_FILE"

# 输出到终端
cat "$OUTPUT_FILE"

echo ""
echo "报告已保存到: $OUTPUT_FILE"

六、告警通知集成：让正确的人在正确的时间收到正确的告警

6.1 Alertmanager配置

Alertmanager负责告警的路由、去重、抑制和静默。以下是我们的生产配置：

# alertmanager-config.yaml
global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.babelbird.com:587'
  smtp_from: 'alertmanager@babelbird.com'
  smtp_auth_username: 'alertmanager@babelbird.com'
  smtp_auth_identity: 'alertmanager@babelbird.com'
  # 从K8s Secret读取SMTP密码
  smtp_auth_password: '${ALERTMANAGER_SMTP_PASSWORD}'

# 告警路由树
route:
  receiver: 'default'
  group_by: ['alertname', 'cluster']
  # 初始等待时间，相同组的告警在此时间内合并
  group_wait: 30s
  # 两次告警通知的间隔
  group_interval: 5m
  # 告警停止后的重复间隔
  repeat_interval: 4h
  # 告警未解决的告警次数上限（超过后停止发送）
  max_alerts: 100
  # 默认路由（未匹配到的告警）
  default receiver: 'default'
  routes:
    # P0告警：严重，立即电话通知
    - match:
        severity: critical
      receiver: 'critical-phone'
      # 严重告警不等待group_wait，立即发出
      group_wait: 0s
      continue: true

    # 存储相关告警：发给存储运维群
    - match:
        alertname: BabelBirdStorageGrowthAnomaly
      receiver: 'storage-team'

    # JVM告警：发给Java团队
    - match:
        alertname: BabelBirdJVMOldGenHighUsage
      receiver: 'java-team'

    # 业务类告警：发给值班SRE
    - match:
        team: babelbird-ops
      receiver: 'sre-oncall'

# 抑制规则：节点NotReady时抑制该节点上所有Pod的Pod级别告警
inhibit_rules:
  - source_match:
      alertname: 'KubernetesNodeNotReady'
      severity: 'critical'
    target_match:
      severity: 'warning'
    # 通过节点名进行关联
    source_match_re:
      node: '(.+)'
    target_match_re:
      node: '(.+)'
    # 相同集群内的抑制
    equal:
      - cluster

  # Prometheus自身告警被抑制（避免监控挂了还发一堆告警）
  - source_match:
      alertname: 'PrometheusScrapeDurationHigh'
      severity: 'critical'
    target_match_re:
      alertname: '.*'
    equal:
      - cluster

# 接收者定义
receivers:
  - name: 'default'
    # 飞书Webhook
    webhook_configs:
      - url: 'http://feishu-alerting.babelbird-app.svc:9097/alert/sdefault'
        send_resolved: true
        max_alerts: 20

  - name: 'critical-phone'
    webhook_configs:
      - url: 'http://feishu-alerting.babelbird-app.svc:9097/alert/critical'
        send_resolved: true
    # 也发邮件
    email_configs:
      - to: 'sre-oncall@babelbird.com'
        headers:
          subject: '【P0告警】{{ range .Alerts }}{{ .Labels.alertname }}{{ end }}'
        send_resolved: true

  - name: 'storage-team'
    webhook_configs:
      - url: 'http://feishu-alerting.babelbird-app.svc:9097/alert/storage'
        send_resolved: true

  - name: 'java-team'
    webhook_configs:
      - url: 'http://feishu-alerting.babelbird-app.svc:9097/alert/java'
        send_resolved: true

  - name: 'sre-oncall'
    webhook_configs:
      - url: 'http://feishu-alerting.babelbird-app.svc:9097/alert/sre'
        send_resolved: true

6.2 飞书告警机器人实现

我们的告警通知通过飞书机器人发送，支持@具体值班人、生成临时会议、告警加急等高级功能：

# feishu_alerting.py
import json
import time
import hmac
import hashlib
import base64
import requests
from urllib.parse import urlencode
from typing import List, Dict, Any, Optional
from datetime import datetime, timedelta
from dataclasses import dataclass, field
from enum import Enum

class AlertSeverity(Enum):
    CRITICAL = "critical"
    WARNING = "warning"
    INFO = "info"

@dataclass
class Alert:
    name: str
    status: str  # firing / resolved
    severity: AlertSeverity
    description: str
    summary: str
    labels: Dict[str, str] = field(default_factory=dict)
    annotations: Dict[str, str] = field(default_factory=dict)
    starts_at: datetime = field(default_factory=datetime.now)
    ends_at: Optional[datetime] = None
    fingerprint: str = ""

    def to_feishu_card(self) -> Dict[str, Any]:
        # 根据告警级别决定卡片颜色和是否加急
        if self.severity == AlertSeverity.CRITICAL:
            color = "red"
            mention_all = True
        elif self.severity == AlertSeverity.WARNING:
            color = "yellow"
            mention_all = False
        else:
            color = "blue"
            mention_all = False

        # 构建标签行
        label_parts = []
        for k, v in self.labels.items():
            if k in ("cluster", "instance", "endpoint", "team"):
                label_parts.append(f"{k}={v}")

        # 构建摘要
        status_emoji = "🔴" if self.status == "firing" else "✅"
        severity_tag = {
            AlertSeverity.CRITICAL: "【P0】",
            AlertSeverity.WARNING: "【P1】",
            AlertSeverity.INFO: "【P2】"
        }.get(self.severity, "")

        card = {
            "msg_type": "interactive",
            "card": {
                "config": {
                    "wide_screen_mode": True,
                    # 严重告警启用加急（强制震动）
                    "enable_forward": True
                },
                "header": {
                    "title": {
                        "tag": "plain_text",
                        "content": f"{status_emoji} {severity_tag}{self.summary}"
                    },
                    "template": color
                },
                "elements": [
                    {
                        "tag": "div",
                        "text": {
                            "tag": "lark_md",
                            "content": f"**告警名称**: `{self.name}`\n"
                                      f"**状态**: {self.status.upper()}\n"
                                      f"**严重程度**: {self.severity.value}\n"
                                      f"**描述**: {self.description}"
                        }
                    }
                ]
            }
        }

        # 添加标签信息
        if label_parts:
            card["card"]["elements"].append({
                "tag": "div",
                "text": {
                    "tag": "lark_md",
                    "content": f"**标签**: `{'` `'.join(label_parts)}`"
                }
            })

        # 添加链接
        if "dashboard_url" in self.annotations:
            card["card"]["elements"].append({
                "tag": "action",
                "actions": [
                    {
                        "tag": "button",
                        "text": {
                            "tag": "plain_text",
                            "content": "📊 查看Dashboard"
                        },
                        "type": "primary",
                        "url": self.annotations.get("dashboard_url", "")
                    },
                    {
                        "tag": "button",
                        "text": {
                            "tag": "plain_text",
                            "content": "📖 查看Runbook"
                        },
                        "type": "default",
                        "url": self.annotations.get("runbook_url", "")
                    }
                ]
            })

        # 添加时间信息
        time_str = self.starts_at.strftime("%Y-%m-%d %H:%M:%S %Z")
        card["card"]["elements"].append({
            "tag": "div",
            "text": {
                "tag": "lark_md",
                "content": f"**触发时间**: {time_str}"
            }
        })

        # 添加@all（严重告警时）
        if mention_all and self.status == "firing":
            card["card"]["elements"].append({
                "tag": "at",
                "at": {
                    "tag": "at",
                    "at_all": True,
                    " ATS": []
                }
            })

        # 解决状态不@all
        if self.status == "resolved":
            card["card"]["elements"].append({
                "tag": "div",
                "text": {
                    "tag": "lark_md",
                    "content": "✅ **告警已解决，感谢处理！**"
                }
            })

        return card


class FeishuAlertClient:
    def __init__(self, webhook_url: str, secret: Optional[str] = None):
        self.webhook_url = webhook_url
        self.secret = secret
        self.session = requests.Session()

    def _sign(self, timestamp: int) -> str:
        """生成飞书签名"""
        if not self.secret:
            return ""
        string_to_sign = f"{timestamp}\n{self.secret}"
        return base64.b64encode(
            hmac.new(
                string_to_sign.encode('utf-8'),
                digestmod=hashlib.sha256
            ).digest()
        ).decode('utf-8')

    def send_alert(self, alert: Alert) -> bool:
        """发送单条告警"""
        card = alert.to_feishu_card()
        return self._send_card(card)

    def send_batch(self, alerts: List[Alert], group_title: str = "告警汇总") -> bool:
        """批量发送告警（合并为一条消息）"""
        if not alerts:
            return True

        elements = []

        for alert in alerts:
            color_map = {
                AlertSeverity.CRITICAL: "red",
                AlertSeverity.WARNING: "yellow",
                AlertSeverity.INFO: "blue"
            }
            color = color_map.get(alert.severity, "blue")

            elements.append({
                "tag": "div",
                "fields": [
                    {"is_short": True, "text": {"tag": "lark_md", "content": f"**告警**\n{alert.summary}"}},
                    {"is_short": True, "text": {"tag": "lark_md", "content": f"**状态**\n{alert.status.upper()}"}},
                    {"is_short": True, "text": {"tag": "lark_md", "content": f"**等级**\n{alert.severity.value}"}},
                    {"is_short": True, "text": {"tag": "lark_md", "content": f"**触发时间**\n{alert.starts_at.strftime('%H:%M:%S')}"}}
                ]
            })
            if alert.annotations.get("dashboard_url"):
                elements.append({
                    "tag": "a",
                    "text": {"tag": "lark_md", "content": "📊 Dashboard"},
                    "href": alert.annotations.get("dashboard_url", "")
                })
            elements.append({"tag": "hr"})

        card = {
            "msg_type": "interactive",
            "card": {
                "config": {"wide_screen_mode": True},
                "header": {
                    "title": {"tag": "plain_text", "content": f"📋 {group_title} ({len(alerts)}条告警)"},
                    "template": "red"
                },
                "element": {
                    "tag": "card",
                    "elements": elements
                }
            }
        }

        # 注意：批量消息需要用不同的结构
        batch_card = {
            "msg_type": "interactive",
            "card": {
                "config": {"wide_screen_mode": True},
                "header": {
                    "title": {"tag": "plain_text", "content": f"📋 {group_title} ({len(alerts)}条)"},
                    "template": "red"
                },
                "elements": elements
            }
        }

        return self._send_card(batch_card)

    def _send_card(self, card: Dict[str, Any]) -> bool:
        """发送卡片消息"""
        timestamp = int(time.time())
        headers = {
            "Content-Type": "application/json"
        }

        payload = card.copy()
        if self.secret:
            payload["timestamp"] = timestamp
            payload["sign"] = self._sign(timestamp)

        try:
            resp = self.session.post(
                self.webhook_url,
                headers=headers,
                json=payload,
                timeout=10
            )
            result = resp.json()
            if result.get("code") == 0 or result.get("StatusCode") == 0:
                return True
            else:
                print(f"Feishu API error: {result}")
                return False
        except Exception as e:
            print(f"Failed to send to Feishu: {e}")
            return False


# 使用示例
if __name__ == "__main__":
    client = FeishuAlertClient(
        webhook_url="https://open.feishu.cn/open-apis/bot/v2/hook/xxx",
        secret="your-secret"
    )

    # 创建并发送告警
    alert = Alert(
        name="BabelBirdHighAPIErrorRate",
        status="firing",
        severity=AlertSeverity.CRITICAL,
        summary="巴别鸟API错误率异常升高",
        description="集群prod-01的API 5xx错误率已达到2.3%，超过1%阈值已持续5分钟",
        labels={"cluster": "prod-01", "team": "babelbird-ops"},
        annotations={
            "dashboard_url": "https://prometheus.babelbird.com/grafana/d/babelbird-overview",
            "runbook_url": "https://runbook.babelbird.com/high-api-error-rate"
        }
    )

    client.send_alert(alert)

七、监控体系的效果与持续改进

7.1 效果量化

监控体系上线18个月后的量化效果：

指标 上线前（Zabbix时代） 上线后 改善幅度
MTTD（平均故障发现时间） 18分钟 47秒 23x
MTTR（平均故障恢复时间） 45分钟 8分钟 5.6x
误报率 35% 8% 4.4x
P0告警响应时间 >30分钟 <5分钟 6x
运维人力投入（日均） 3.5人时 0.8人时 4.4x

7.2 持续改进机制

每周指标审查
每周一运维团队花30分钟审查上周告警记录，分析：

  • 有没有漏报？（故障发生了但没有告警）
  • 有没有误报？（告警了但其实没问题）
  • 告警是否及时？
  • 处置速度是否正常？

每月Dashboardreview
每月检查各Dashboard的使用情况，删除无人查看的Panel，补充新指标的Panel。Dashboard也是技术债，不维护就会过期。

每季度监控策略评审
结合季度故障复盘，评估监控策略是否需要调整。故障模式在变，监控策略也要跟着变。

八、踩坑清单汇总

把文中提到的所有坑整理成清单，方便快速查阅：

序号 场景 问题 解决方案
1 Grafana Dashboard持久化 从ConfigMap加载的Dashboard只读，无法在UI修改 provider.allowUiUpdates: true；采用GitOps策略：UI编辑→导出→更新ConfigMap
2 ServiceMonitor不生效 新exporter在babelbird-new命名空间，Prometheus拉取不到 Prometheus CRD默认只匹配monitoring命名空间；添加namespace白名单或标签匹配
3 JMX Exporter指标噪音 全量MBean暴露几百个无用指标，存储压力大 自研业务层Exporter，只暴露业务关心的核心指标
4 Helm安装Prometheus超时 镜像拉取量大，默认timeout不够 --timeout 10m --wait --atomic --cleanup-on-fail
5 Alertmanager告警风暴 同类告警短时间大量涌入 配置group_wait、group_interval；启用抑制规则（inhibit_rules）
6 日志收集效率低 SSH手动拖日志，10+台机器就崩溃 一键脚本collect_logs.sh，支持namespace/时间/关键词过滤、并行收集、自动压缩
7 告警无区分度 P0/P1/P2告警混在一起，值班人员不知道优先处理哪个 Alertmanager路由按severity标签分派不同接收者；严重告警group_wait: 0s立即发出
8 JVM指标缺少业务语义 GC次数、堆内存都是通用指标，排查具体问题帮助有限 自定义Exporter暴露业务指标（上传成功率、存储用量、预览转换状态）带业务标签

九、写在最后

监控体系的建设没有终点。随着巴别鸟业务规模增长、架构演进，监控也要跟着变。Prometheus + Grafana + 业务Exporter + 自动化脚本这套组合拳，让我们的运维从"救火"走向了"预防"，这是最重要的转变。

如果你正在为企业的可观测性建设头疼，欢迎参考本文的思路。核心建议只有一条：先想清楚你要监控什么（而不是有什么监控工具），然后选择最适合的方案去实现。监控工具本身不是目的，快速发现和定位问题才是。

相关资源

  • kube-prometheus-stack: https://github.com/prometheus-community/helm-charts
  • Prometheus Operator: https://prometheus-operator.dev/
  • Grafana Dashboard模板市场: https://grafana.com/grafana/dashboards/
  • 巴别鸟企业云盘: https://www.babelbird.com

本文基于巴别鸟生产环境实际经验编写，所有代码和配置均经过验证。如有问题或建议，欢迎通过技术团队联系方式交流。

# 运维 # 自动化 # prometheus # 安全 # 网络 # 云计算
Logo
https://edu.csdn.net/learn/39067/627173?utm_source=2019755004

汇聚全球AI编程工具，助力开发者即刻编程。

更多推荐

cover

鸿蒙PC日常编码agent推荐，deveco code竟查无此人

avatar AI编程社区

别卷模型了！OpenAI 工程师都在偷偷用的“Harness Engineering“，才是 AI 编程的终极杀器

文章摘要： OpenAI工程师提出“约束工程”（HarnessEngineering），通过设计明确的约束框架（如角色设定、输出格式、渐进式推理等），引导现有AI模型生成更精准、可控的输出，而非一味追求更大参数量。相比更换模型，该方法成本更低、确定性更高，能有效减少幻觉。实践模式包括模板化生成、渐进式推理链和多轮校验循环。开发者可通过结构化prompt模板和反馈闭环快速落地，将AI能力最大化。这一

avatar AI编程社区
cover

Codex并不难，难的是知道什么时候不该用

avatar AI编程社区