Kubernetes 蓝绿发布与金丝雀发布

眷蓝天

388人浏览 · 2026-05-14 18:53:16

眷蓝天 · 2026-05-14 18:53:16 发布

1. 简介与核心作用

灰度发布通过控制流量分配验证新版本，是降低发布风险的核心手段。

1.1 发布策略对比

表格

策略	原理	资源开销	回滚速度	适用场景
滚动更新	逐步替换旧 Pod	低	快	无状态服务快速迭代
蓝绿发布	双套环境，切换 selector	高	秒级	有状态服务、需完整验证
金丝雀发布	灰度流量逐步扩大	低	较快	高风险功能、特性验证

1.2 架构概览

plaintext

┌────────────────────────────────────────────────────────────────┐
│  滚动更新: [V1 Pod] → [V1 Pod] → [V2 Pod 逐步替换]              │
│  蓝绿发布: [Blue环境] ──→ [Green环境] ──selector切换──→ 流量   │
│  金丝雀: [V1×9] ──90%──→ [V2×1] ──10%──→ (逐步扩大)            │
└────────────────────────────────────────────────────────────────┘

2. 蓝绿发布详解

2.1 原理与架构

蓝绿发布维持两套完整环境，流量切换通过修改 Service Selector 实现。

plaintext

┌────────────────────────────────────────────────────────────────┐
│  ┌────────────┐      ┌────────────┐      ┌──────────┐         │
│  │ Blue (V1)  │      │ Green (V2) │      │ Service  │         │
│  │ track:blue │      │ track:green│ ←──  │ selector │         │
│  └────────────┘      └────────────┘      │ track:?  │         │
│                                          └────┬─────┘         │
│                                               ↓                │
│                                         流量 100%              │
└────────────────────────────────────────────────────────────────┘

2.2 完整 YAML 示例

yaml

# 蓝绿发布完整配置
---
# Blue 环境 - 当前版本
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-blue
  labels:
    app: myapp
    track: blue
spec:
  replicas: 3
  strategy:
    type: Recreate  # 蓝绿必须用 Recreate
  selector:
    matchLabels:
      app: myapp
      track: blue
  template:
    metadata:
      labels:
        app: myapp
        track: blue
        version: v1.0.0
    spec:
      containers:
      - name: myapp
        image: myapp:v1.0.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 5
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 15
---
# Green 环境 - 新版本
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-green
  labels:
    app: myapp
    track: green
spec:
  replicas: 3
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: myapp
      track: green
  template:
    metadata:
      labels:
        app: myapp
        track: green
        version: v2.0.0
    spec:
      containers:
      - name: myapp
        image: myapp:v2.0.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 5
---
# Service - 切换 selector 改变流量
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: myapp
    track: blue  # 改为 green 切换流量

2.3 切换脚本

bash

#!/bin/bash
# 蓝绿切换: ./bluegreen-switch.sh <blue|green>
set -e
NS="${NAMESPACE:-default}"
SVC="${SERVICE_NAME:-myapp}"
TRACK="${1:-green}"
[[ "$TRACK" == "blue" || "$TRACK" == "green" ]] || exit 1
echo "Switching to $TRACK..."
READY=$(kubectl get deployment myapp-$TRACK -n $NS -o jsonpath='{.status.readyReplicas}')
REPLICAS=$(kubectl get deployment myapp-$TRACK -n $NS -o jsonpath='{.spec.replicas}')
[[ "$READY" == "$REPLICAS" ]] || { echo "Not ready ($READY/$REPLICAS)"; exit 1; }
kubectl patch service $SVC -n $NS --type='json' \
    -p="[{'op': 'replace', 'path': '/spec/selector/track', 'value': '$TRACK'}]"
echo "✓ Switched to $TRACK"

3. 金丝雀发布详解

3.1 原理与架构

金丝雀发布通过控制进入新版本的流量比例，逐步验证后全量切换。

plaintext

┌────────────────────────────────────────────────────────────────┐
│  阶段1 10%: [V1×9] ───90%──→ [V2×1] ──10%──→ 监控验证          │
│  阶段2 50%: [V1×5] ───50%──→ [V2×5] ──50%──→ 全面验证         │
│  阶段3 100%:[V2×10] ───────────────100%────────────────→ 完成 │
│                                                                │
│  控制方式: 原生K8S(副本比例) | Ingress Nginx | Istio | Argo    │
└────────────────────────────────────────────────────────────────┘

3.2 实现方式对比

表格

方式	流量控制	配置	依赖	场景
原生 K8S	Pod 副本比例	低	无	简单场景
Ingress Nginx	权重/Header/Cookie	中	nginx-ingress	HTTP 入口
Istio	任意比例	高	Istio	精细流量管理
Argo Rollouts	渐进+自动分析	中	Argo CD	GitOps

3.3 原生 K8S 实现

yaml

# 原生 K8S 金丝雀 - 副本比例 ≈ 流量比例
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-v1
spec:
  replicas: 9  # 75%
  selector:
    matchLabels:
      app: myapp
      version: v1
  template:
    metadata:
      labels:
        app: myapp
        version: v1
    spec:
      containers:
      - name: myapp
        image: myapp:v1.0.0
        ports:
        - containerPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-v2
spec:
  replicas: 3  # 25%
  selector:
    matchLabels:
      app: myapp
      version: v2
  template:
    metadata:
      labels:
        app: myapp
        version: v2
    spec:
      containers:
      - name: myapp
        image: myapp:v2.0.0
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp  # 不限定 version
  ports:
  - port: 80
    targetPort: 8080

3.4 Ingress Nginx 金丝雀

yaml

# Ingress Nginx 金丝雀 - 权重/Header/Cookie
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp-v1
            port:
              number: 80
---
# 金丝雀 Ingress - 权重模式
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "20"  # 20%
spec:
  ingressClassName: nginx
  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp-v2
            port:
              number: 80
---
# Header 模式 - 内测用户
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-canary-header
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "X-Canary"
    nginx.ingress.kubernetes.io/canary-by-header-value: "always"
spec:
  ingressClassName: nginx
  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp-v2
            port:
              number: 80

注解说明: canary 启用 | canary-weight 权重 | canary-by-header / canary-by-cookie 条件匹配

3.5 Istio VirtualService 金丝雀

yaml

# Istio 金丝雀
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-v1
spec:
  replicas: 9
  selector:
    matchLabels:
      app: myapp
      version: v1
  template:
    metadata:
      labels:
        app: myapp
        version: v1
    spec:
      containers:
      - name: myapp
        image: myapp:v1.0.0
        ports:
        - containerPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
      version: v2
  template:
    metadata:
      labels:
        app: myapp
        version: v2
    spec:
      containers:
      - name: myapp
        image: myapp:v2.0.0
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: myapp
---
# DestinationRule - 版本子集
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: myapp
spec:
  host: myapp
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
---
# VirtualService - 流量分配
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp
spec:
  hosts:
  - myapp
  http:
  - route:
    - destination:
        host: myapp
        subset: v1
      weight: 90
    - destination:
        host: myapp
        subset: v2
      weight: 10

3.6 Argo Rollouts 渐进式发布

yaml

# Argo Rollouts - 声明式金丝雀 + 自动分析
# 安装: kubectl apply -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml
---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: myapp
spec:
  replicas: 10
  strategy:
    canary:
      steps:
      - setWeight: 10
      - pause: {}          # 等待确认
      - setWeight: 50
      - pause: {duration: 10}  # 等待 10 分钟
      - setWeight: 100
      analysis:
        templates:
        - templateName: myapp-analysis
        args:
        - name: service-name
          value: myapp
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:v2.0.0
        ports:
        - containerPort: 8080
---
# 自动分析模板
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: myapp-analysis
spec:
  args:
  - name: service-name
  metrics:
  - name: error-rate
    interval: 1m
    successCondition: result[0] < 0.05
    failureLimit: 3
    provider:
      prometheus:
        address: http://prometheus:9090
        query: |
          sum(rate(http_requests_total{service="{{args.service-name}}",status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{service="{{args.service-name}}"}[5m]))

4. 实战场景

4.1 蓝绿发布完整流程

yaml

# bluegreen-full.yaml - 可运行的完整示例
---
apiVersion: v1
kind: Namespace
metadata:
  name: production
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-config
  namespace: production
data:
  nginx.conf: |
    server {
      listen 8080;
      location / { return 200 'Version 1.0.0\n'; add_header Content-Type text/plain; }
      location /health { return 200 'OK\n'; }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-blue
  namespace: production
spec:
  replicas: 3
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: myapp
      track: blue
  template:
    metadata:
      labels:
        app: myapp
        track: blue
        version: v1.0.0
    spec:
      containers:
      - name: myapp
        image: nginx:1.25-alpine
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: config
          mountPath: /etc/nginx/conf.d
        readinessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 3
      volumes:
      - name: config
        configMap:
          name: myapp-config
---
apiVersion: v1
kind: Service
metadata:
  name: myapp
  namespace: production
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: myapp
    track: blue

bash

# 蓝绿发布执行流程
# 1. 部署初始版本
kubectl apply -f bluegreen-full.yaml
# 2. 部署新版本 Green 环境
kubectl create -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-config-green
  namespace: production
data:
  nginx.conf: |
    server {
      listen 8080;
      location / { return 200 'Version 2.0.0\n'; add_header Content-Type text/plain; }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-green
  namespace: production
spec:
  replicas: 3
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: myapp
      track: green
  template:
    metadata:
      labels:
        app: myapp
        track: green
        version: v2.0.0
    spec:
      containers:
      - name: myapp
        image: nginx:1.25-alpine
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: config
          mountPath: /etc/nginx/conf.d
        readinessProbe:
          tcpSocket:
            port: 8080
      volumes:
      - name: config
        configMap:
          name: myapp-config-green
EOF
# 3. 验证并切换
kubectl rollout status deployment myapp-green -n production
kubectl patch service myapp -n production -p '{"spec":{"selector":{"track":"green"}}}'
# 4. 验证
kubectl get pods -n production -l track=green
# 5. 清理旧版本
kubectl delete deployment myapp-blue -n production
kubectl delete configmap myapp-config -n production

4.2 金丝雀监控告警

yaml

# Prometheus 告警规则
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: myapp-alerts
spec:
  groups:
  - name: canary-alerts
    rules:
    - alert: CanaryHighErrorRate
      expr: |
        sum(rate(http_requests_total{version="v2",status=~"5.."}[5m]))
        / sum(rate(http_requests_total{version="v2"}[5m])) > 0.01
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: "Canary error rate > 1%"
    - alert: CanaryHighLatency
      expr: |
        histogram_quantile(0.99,
          sum(rate(http_request_duration_seconds_bucket{version="v2"}[5m])) by (le)
        ) > 2
      for: 5m
      labels:
        severity: warning

5. 常用操作命令

bash

# 蓝绿发布
kubectl patch service myapp -n production -p '{"spec":{"selector":{"track":"green"}}}'
kubectl get service myapp -n production -o jsonpath='{.spec.selector}'
kubectl delete deployment myapp-blue -n production
# 金丝雀 - 原生 K8S
kubectl scale deployment myapp-v2 --replicas=3
kubectl get pods -l app=myapp -L version
# 金丝雀 - Ingress Nginx
kubectl annotate ingress myapp-canary nginx.ingress.kubernetes.io/canary-weight="30" --overwrite
# 金丝雀 - Istio
kubectl patch virtualservice myapp -p '{"spec":{"http":[{"route":[{"dest":"myapp","subset":"v1","w":70},{"dest":"myapp","subset":"v2","w":30}]}]}}'
# 金丝雀 - Argo Rollouts
kubectl argo rollouts promote myapp -n production
kubectl argo rollouts abort myapp -n production
kubectl argo rollouts get rollout myapp -n production -w
# 发布状态检查
kubectl get deploy,pods -n production -l app=myapp -o wide

6. 常见问题与排查

6.1 蓝绿切换后流量未生效

bash

# 排查步骤
kubectl get service myapp -n production -o jsonpath='{.spec.selector}'
kubectl get endpoints myapp -n production
kubectl get pods -n production -l app=myapp,track=green
# 常见原因与解决
# Pod 未就绪: kubectl rollout status deployment myapp-green
# selector 不匹配: 确认 label 一致
# 代理缓存: 清理 CDN/Nginx 缓存

6.2 金丝雀流量比例不符

bash

# 原生 K8S: 确认副本数
kubectl get deploy -l app=myapp -o jsonpath='{range .items[*]}{.metadata.name}: {.spec.replicas}{"\n"}{end}'
# Ingress Nginx: 检查注解
kubectl describe ingress myapp-canary | grep -A 5 Annotations
# Istio: 检查路由
kubectl get virtualservice myapp -o yaml | grep -A 10 weight
kubectl get destinationrule myapp -n production -o yaml
# 检查会话亲和
kubectl get svc myapp -o jsonpath='{.spec.sessionAffinity}'

6.3 Pod 不健康导致发布中断

bash

# 查看事件和日志
kubectl describe pod <pod-name> -n production | tail -50
kubectl logs <pod-name> -n production --previous
# 常见错误:
# CrashLoopBackOff - 应用启动失败，检查依赖/配置
# ImagePullBackOff - 镜像拉取失败，检查镜像地址
# OOMKilled - 内存超限，调高 memory limit
# 紧急回滚
kubectl patch service myapp -n production -p '{"spec":{"selector":{"track":"blue"}}}'
kubectl scale deployment myapp-green -n production --replicas=0

7. 最佳实践

7.1 策略选型

plaintext

┌────────────────────────────────────────────────────────────────┐
│  Bug 修复 / 小改动 ───→ 滚动更新                                  │
│  新功能 / 中等风险 ───→ 金丝雀发布 (10% → 50% → 100%)            │
│  大版本 / 高风险 ─────→ 蓝绿发布 (双环境 + 秒级回滚)               │
│  有状态服务 ─────────→ 蓝绿发布 (必须)                            │
└────────────────────────────────────────────────────────────────┘

7.2 发布前检查

bash

# 资源配额
kubectl get resourcequota -n production
# 依赖服务
kubectl get svc -n production
# 镜像可用性
kubectl run test --image=myapp:v2.0.0 --rm -it --restart=Never -- echo OK 2>&1 | head -3

7.3 监控指标

表格

类型	指标	阈值
错误率	HTTP 5xx	< 1%
延迟	P99	< 1s
可用性	成功率	> 99.9%

7.4 GitOps 流水线

yaml

# CI/CD 金丝雀示例
stages:
  - canary-deploy
  - promote
canary-deploy:
  script:
    - kubectl set image deployment/myapp-v2 app=${IMAGE}:${CI_COMMIT_SHA} -n production
    - kubectl scale deployment myapp-v2 --replicas=1 -n production
    - sleep 300  # 5分钟观察
    - kubectl scale deployment myapp-v2 --replicas=5 -n production
promote:
  script:
    - kubectl scale deployment myapp-v1 --replicas=0 -n production
    - kubectl scale deployment myapp-v2 --replicas=10 -n production
  when: manual

7.5 关键要点

表格

阶段	要点
选型	小改动用滚动，有状态用蓝绿，新功能用金丝雀
准备	监控先行，健康检查完整，回滚方案就绪
部署	先小比例，观察指标，逐步扩大
验证	错误率、延迟、业务指标三重验证
全量	确认无误后删除旧版本
回滚	任何异常立即回滚，不要犹豫

附录: 快速命令

bash

# 蓝绿切换
kubectl patch service myapp -n production -p '{"spec":{"selector":{"track":"green"}}}'
# 金丝雀 - 原生 K8S
kubectl scale deployment myapp-v2 --replicas=3
# 金丝雀 - Ingress Nginx
kubectl annotate ingress myapp-canary nginx.ingress.kubernetes.io/canary-weight="30" --overwrite
# 金丝雀 - Istio
kubectl patch virtualservice myapp -p '{"spec":{"http":[{"route":[{"dest":"myapp","subset":"v1","w":70},{"dest":"myapp","subset":"v2","w":30}]}]}}'
# Argo Rollouts
kubectl argo rollouts promote myapp -n production
# 查看状态
kubectl get pods -l app=myapp -L version
# 紧急回滚
kubectl patch service myapp -n production -p '{"spec":{"selector":{"track":"blue"}}}'