使用 RKE 构建企业生产级 Kubernetes 集群（七）：etcd 节点管理与 Nginx 应用部署验证

摘要：本文深入讲解 etcd 分布式存储的核心原理与运维最佳实践，涵盖 etcd 架构设计、性能调优、备份恢复、故障处理等关键技术，并通过完整的 Nginx 应用部署实战验证 Kubernetes 集群的可用性。通过详细的配置示例、性能基准测试和故障恢复案例，帮助读者全面掌握 etcd 运维与应用部署的核心技能。

关键词：etcd；分布式存储；备份恢复；性能调优；Nginx 部署；应用验证；Kubernetes

---

1 引言

1.1 etcd 在 Kubernetes 中的核心地位

etcd 是 Kubernetes 集群的"大脑"，负责存储集群的所有状态数据。根据生产环境统计数据：

• 99.99% 可用性要求：etcd 故障将导致整个集群不可用
• 性能瓶颈 80%：Kubernetes API 操作延迟主要来自 etcd
• 数据零丢失：etcd 数据丢失意味着集群状态完全丢失

1.2 应用验证的重要性

部署应用是验证 Kubernetes 集群可用性的最终标准：

• 网络验证：Pod 间通信、Service 访问、Ingress 路由
• 存储验证：PersistentVolume 挂载、数据持久化
• 调度验证：Pod 调度策略、资源分配
• 高可用验证：故障转移、自动恢复

1.3 本文技术要点

• etcd 架构深度解析：Raft 共识算法、数据模型、性能特性
• etcd 性能调优：磁盘优化、参数配置、监控指标
• etcd 备份恢复：快照备份、灾难恢复、数据迁移
• etcd 故障处理：常见问题诊断、集群重建、数据修复
• Nginx 应用部署：Deployment、Service、Ingress 完整配置
• 应用高可用验证：负载均衡、健康检查、自动扩缩容

---

2 etcd 架构深度解析

2.1 etcd 核心架构

2.1.1 数据模型

etcd 数据模型（B+ 树结构）:
┌─────────────────────────────────────┐
│          Root (Revision: 1024)       │
└─────────────────────────────────────┘
              ↓
┌─────────────────────────────────────┐
│   /registry                          │
│   ├── /pods                          │
│   │   ├── /default/nginx-pod-1      │
│   │   └── /default/nginx-pod-2      │
│   ├── /deployments                   │
│   │   └── /default/nginx-deployment  │
│   ├── /services                      │
│   └── /configmaps                    │
└─────────────────────────────────────┘

关键概念：

• Key-Value 存储：所有数据以键值对形式存储
• Revision：全局递增版本号，每次写操作递增
• Lease：租约机制，实现 TTL 功能
• Watch：监听机制，实时通知变更

2.1.2 Raft 共识算法

Leader 选举流程：

1. 初始状态：所有节点为 Follower
2. 选举超时（150-300ms 随机）：Follower 变为 Candidate
3. 请求投票：Candidate 向其他节点发送 RequestVote
4. 获得多数票：成为 Leader
5. 心跳维持：Leader 定期发送心跳给 Follower
6. 故障检测：Follower 超时未收到心跳，重新选举

数据复制流程：

Client → Leader → AppendEntries → Followers
          ↓                        ↓
      写入本地 WAL            写入本地 WAL
          ↓                        ↓
      提交日志 ← 多数确认 ←      发送确认
          ↓
      应用状态机
          ↓
      返回客户端

2.2 etcd 性能特性

2.2.1 性能瓶颈分析

写操作延迟组成：

总延迟 = 网络延迟 + WAL 写入 + Raft 复制 + 提交确认

典型值（10GbE 网络）:
- 网络延迟：0.5ms
- WAL 写入：2-5ms（取决于磁盘）
- Raft 复制：1-2ms
- 提交确认：0.5ms
总延迟：4-9ms

读操作延迟组成：

线性化读：网络 + 共识（需要 Leader 确认）
串行读：网络 + 本地读取
本地读：仅网络延迟

2.2.2 性能基准数据

测试环境：

• 硬件：8 核 CPU, 16GB 内存，NVMe SSD
• 网络：10GbE
• etcd 版本：3.5.6
• 节点数：3

测试结果：

操作类型	QPS	P50 延迟	P99 延迟
写（1KB）	8,000	3.2ms	8.5ms
线性化读	50,000	0.8ms	2.1ms
串行读	100,000	0.5ms	1.5ms
本地读	200,000	0.3ms	0.8ms

---

3 etcd 性能调优

3.1 磁盘优化

3.1.1 磁盘选择

磁盘性能要求：

# 使用 fio 测试磁盘性能
fio --name=seqwrite --ioengine=libaio --iodepth=32 \
    --rw=write --bs=1M --direct=1 --size=4G \
    --numjobs=1 --runtime=60 --group_reporting

fio --name=randwrite --ioengine=libaio --iodepth=256 \
    --rw=randwrite --bs=4k --direct=1 --size=4G \
    --numjobs=1 --runtime=60 --group_reporting

性能标准：

• IOPS：> 5000（随机写）
• 吞吐：> 500MB/s（顺序写）
• 延迟：< 10ms（P99）

推荐配置：

• 生产环境：NVMe SSD（Intel Optane, Samsung 980 Pro）
• 中型集群：SATA SSD（Samsung 870 EVO）
• 开发测试：SATA HDD（仅功能验证）

3.1.2 文件系统优化

# /etc/fstab 配置（NVMe SSD）
/dev/nvme0n1  /var/lib/etcd  xfs  noatime,nodiratime,logbufs=8,logbsize=256k  0 0

# 挂载选项说明:
# noatime: 不更新访问时间
# nodiratime: 不更新目录访问时间
# logbufs=8: 增加日志缓冲区
# logbsize=256k: 日志缓冲区大小

# 应用配置
mount -o remount /var/lib/etcd

# 验证挂载选项
mount | grep etcd

3.2 etcd 参数调优

3.2.1 核心参数配置

# RKE cluster.yml etcd 配置
services:
  etcd:
    extra_args:
      # 选举超时（毫秒）
      election-timeout: "5000"
      
      # 心跳间隔（毫秒）
      heartbeat-interval: "500"
      
      # 自动压缩保留历史
      auto-compaction-retention: "8"
      
      # 后端存储配额（8GB）
      quota-backend-bytes: "8589934592"
      
      # 最大请求大小（1.5MB）
      max-request-bytes: "1572864"
      
      # 快照计数
      snapshot-count: "10000"
      
      # 预缓存
      experimental-initial-corrupt-check: "true"
      
      # 检查间隔
      experimental-corrupt-check-time: "240m"
      
      # 最大并发流
      max-concurrent-streams: "1000"
      
    # 日志配置
    extra_env:
      - ETCD_DEBUG=true
      - ETCD_TRACING=true

3.2.2 内存管理

# etcd 内存使用组成:
# 1. 后端存储（BoltDB）：占用主要内存
# 2. 缓存：最近访问的数据
# 3. Go Runtime：GC 开销

# 内存限制配置
# /etc/systemd/system/etcd.service.d/memory.conf
[Service]
MemoryLimit=8G
MemoryHigh=6G
MemorySwapLimit=0

# 重启服务
systemctl daemon-reload
systemctl restart etcd

3.3 网络优化

# 内核参数优化
cat >> /etc/sysctl.d/99-etcd.conf <<EOF
# 增加连接队列
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

# 增加端口范围
net.ipv4.ip_local_port_range = 1024 65535

# 启用 TCP 快速回收
net.ipv4.tcp_tw_reuse = 1

# 增加缓冲区
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 12582912 16777216
net.ipv4.tcp_wmem = 4096 12582912 16777216

# 应用配置
sysctl --system
EOF

---

4 etcd 备份与恢复

4.1 备份策略

4.1.1 定期快照备份

# RKE 自动备份配置
services:
  etcd:
    snapshot: true
    backup_config:
      enabled: true
      interval_hours: 12      # 每 12 小时备份
      retention: 6            # 保留 6 个备份
      s3backupconfig:
        access_key: AKIAIOSFODNN7EXAMPLE
        secret_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
        bucket_name: k8s-etcd-backups
        endpoint: s3.amazonaws.com
        region: us-east-1

# 手动创建快照
rke etcd snapshot-save \
  --config cluster.yml \
  --name etcd-snapshot-$(date +%Y%m%d-%H%M%S)

# 列出快照
rke etcd snapshot-list --config cluster.yml

# 下载快照到本地
rke etcd snapshot-pull \
  --config cluster.yml \
  --name etcd-snapshot-20240115-103000 \
  --target /backup/etcd/

4.1.2 etcdctl 快照

# 创建快照
ETCDCTL_API=3 etcdctl snapshot save snapshot.db \
  --endpoints=https://192.168.1.101:2379 \
  --cacert=/etc/kubernetes/ssl/kube-ca.pem \
  --cert=/etc/kubernetes/ssl/kube-node.pem \
  --key=/etc/kubernetes/ssl/kube-node-key.pem

# 验证快照
ETCDCTL_API=3 etcdctl snapshot status snapshot.db \
  --write-out=table

# 输出示例:
# +----------+----------+------------+------------+
#|   KEY    | REVISION | TOTAL KEYS | TOTAL SIZE |
# +----------+----------+------------+------------+
# | snapshot.db | 1048576 | 15234     | 256 MB     |
# +----------+----------+------------+------------+

4.2 灾难恢复

4.2.1 单节点故障恢复

# 1. 停止故障节点
ssh ubuntu@192.168.1.101 "docker stop etcd"

# 2. 移除故障节点
ETCDCTL_API=3 etcdctl member remove 422a64f5b3b75d2c \
  --endpoints=https://192.168.1.102:2379 \
  --cacert=/etc/kubernetes/ssl/kube-ca.pem \
  --cert=/etc/kubernetes/ssl/kube-node.pem \
  --key=/etc/kubernetes/ssl/kube-node-key.pem

# 3. 清理旧数据
ssh ubuntu@192.168.1.101 "rm -rf /var/lib/etcd/*"

# 4. 添加新节点
ETCDCTL_API=3 etcdctl member add etcd1-new \
  --peer-urls=https://192.168.1.101:2380 \
  --endpoints=https://192.168.1.102:2379 \
  --cacert=/etc/kubernetes/ssl/kube-ca.pem \
  --cert=/etc/kubernetes/ssl/kube-node.pem \
  --key=/etc/kubernetes/ssl/kube-node-key.pem

# 输出:
# ETCD_NAME="etcd1-new"
# ETCD_INITIAL_CLUSTER="etcd2=https://192.168.1.102:2380,etcd3=https://192.168.1.103:2380,etcd1-new=https://192.168.1.101:2380"
# ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.101:2380"
# ETCD_INITIAL_CLUSTER_STATE="existing"

# 5. 启动新节点
ssh ubuntu@192.168.1.101 <<'EOF'
docker run -d --restart=unless-stopped \
  --name etcd \
  --net=host \
  -v /var/lib/etcd:/var/lib/rancher/etcd \
  -e ETCD_NAME=etcd1-new \
  -e ETCD_INITIAL_CLUSTER=etcd2=https://192.168.1.102:2380,etcd3=https://192.168.1.103:2380,etcd1-new=https://192.168.1.101:2380 \
  -e ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.1.101:2380 \
  -e ETCD_INITIAL_CLUSTER_STATE=existing \
  rancher/mirrored-coreos-etcd:v3.5.6 \
  etcd --name=etcd1-new \
    --advertise-client-urls=https://192.168.1.101:2379 \
    --initial-advertise-peer-urls=https://192.168.1.101:2380 \
    --listen-client-urls=https://192.168.1.101:2379 \
    --listen-peer-urls=https://192.168.1.101:2380 \
    --initial-cluster=etcd2=https://192.168.1.102:2380,etcd3=https://192.168.1.103:2380,etcd1-new=https://192.168.1.101:2380 \
    --initial-cluster-state=existing
EOF

# 6. 验证节点加入
ETCDCTL_API=3 etcdctl member list \
  --endpoints=https://192.168.1.102:2379 \
  --cacert=/etc/kubernetes/ssl/kube-ca.pem \
  --cert=/etc/kubernetes/ssl/kube-node.pem \
  --key=/etc/kubernetes/ssl/kube-node-key.pem

4.2.2 数据恢复（从快照）

# 1. 停止所有 etcd 节点
for node in 192.168.1.101 192.168.1.102 192.168.1.103; do
  ssh ubuntu@$node "docker stop etcd"
done

# 2. 备份现有数据（以防万一）
for node in 192.168.1.101 192.168.1.102 192.168.1.103; do
  ssh ubuntu@$node "mv /var/lib/etcd /var/lib/etcd.backup.$(date +%Y%m%d)"
  ssh ubuntu@$node "mkdir -p /var/lib/etcd"
done

# 3. 解压快照到第一个节点
tar -xzf etcd-snapshot-20240115-103000.db.gz -C /tmp/

# 4. 恢复数据到第一个节点
ssh ubuntu@192.168.1.101 <<'EOF'
ETCDCTL_API=3 etcdctl snapshot restore /tmp/etcd-snapshot.db \
  --data-dir=/var/lib/etcd \
  --name=etcd1 \
  --initial-cluster=etcd1=https://192.168.1.101:2380,etcd2=https://192.168.1.102:2380,etcd3=https://192.168.1.103:2380 \
  --initial-advertise-peer-urls=https://192.168.1.101:2380
EOF

# 5. 复制数据到其他节点
for node in 192.168.1.102 192.168.1.103; do
  scp -r /var/lib/etcd/* ubuntu@$node:/var/lib/etcd/
done

# 6. 启动所有节点
for node in 192.168.1.101 192.168.1.102 192.168.1.103; do
  ssh ubuntu@$node "docker start etcd"
done

# 7. 验证集群健康
ETCDCTL_API=3 etcdctl endpoint health \
  --endpoints=https://192.168.1.101:2379,https://192.168.1.102:2379,https://192.168.1.103:2379 \
  --cacert=/etc/kubernetes/ssl/kube-ca.pem \
  --cert=/etc/kubernetes/ssl/kube-node.pem \
  --key=/etc/kubernetes/ssl/kube-node-key.pem

---

5 etcd 监控与告警

5.1 关键监控指标

# Prometheus 监控指标
# etcd 服务器指标
- etcd_server_has_leader: etcd 是否有 Leader
- etcd_server_leader_changes_seen_total: Leader 变更次数
- etcd_server_proposals_failed_total: 提案失败数
- etcd_server_proposals_applied_total: 提案应用数
- etcd_server_proposals_committed_total: 提案提交数
- etcd_server_proposals_pending_total: 待处理提案

# 性能指标
- etcd_disk_backend_commit_duration_seconds: 磁盘提交延迟
- etcd_disk_wal_fsync_duration_seconds: WAL 同步延迟
- etcd_network_peer_round_trip_time_seconds: 节点间 RTT
- etcd_grpc_server_msg_sent_total: gRPC 发送消息数
- etcd_grpc_server_msg_received_total: gRPC 接收消息数

# 资源指标
- etcd_mvcc_db_total_size_in_bytes: 数据库大小
- etcd_mvcc_db_total_size_in_use_in_bytes: 实际使用大小
- etcd_debugging_mvcc_keys_total: 总 Key 数
- etcd_debugging_mvcc_watch_stream_total: Watch 流数量

5.2 告警规则

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: etcd-alerts
  namespace: monitoring
spec:
  groups:
  - name: etcd
    rules:
    - alert: etcdHighNumberOfFailedGRPCRequests
      expr: sum(rate(etcd_grpc_server_msg_sent_total{grpc_code!="OK"}[5m])) > 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "etcd gRPC 请求失败"
        description: "etcd 集群 gRPC 请求失败率过高"
    
    - alert: etcdHighNumberOfFailedProposals
      expr: rate(etcd_server_proposals_failed_total[5m]) > 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "etcd 提案失败"
        description: "etcd 集群提案失败"
    
    - alert: etcdHighNumberOfLeaderChanges
      expr: rate(etcd_server_leader_changes_seen_total[10m]) > 2
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "etcd Leader 频繁变更"
        description: "etcd 集群在 10 分钟内发生多次 Leader 变更"
    
    - alert: etcdDatabaseHighSize
      expr: etcd_mvcc_db_total_size_in_bytes / etcd_server_quota_backend_bytes > 0.8
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "etcd 数据库过大"
        description: "etcd 数据库使用率超过 80%"
    
    - alert: etcdDiskSlow
      expr: histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) > 0.025
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "etcd 磁盘延迟过高"
        description: "etcd 磁盘 P99 延迟超过 25ms"

---

6 Nginx 应用部署实战

6.1 Deployment 配置

# nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: default
  labels:
    app: nginx
    version: v1.21
spec:
  replicas: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
        version: v1.21
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9113"
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: nginx
              topologyKey: kubernetes.io/hostname
      containers:
      - name: nginx
        image: nginx:1.21.6
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        resources:
          requests:
            cpu: 250m
            memory: 256Mi
          limits:
            cpu: 500m
            memory: 512Mi
        livenessProbe:
          httpGet:
            path: /healthz
            port: 80
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3
          successThreshold: 1
        readinessProbe:
          httpGet:
            path: /ready
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 3
          successThreshold: 1
        volumeMounts:
        - name: nginx-config
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
        - name: nginx-logs
          mountPath: /var/log/nginx
        - name: nginx-cache
          mountPath: /var/cache/nginx
      volumes:
      - name: nginx-config
        configMap:
          name: nginx-config
          items:
          - key: nginx.conf
            path: nginx.conf
      - name: nginx-logs
        emptyDir: {}
      - name: nginx-cache
        emptyDir: {}
      tolerations:
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"

6.2 ConfigMap 配置

# nginx-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
  namespace: default
data:
  nginx.conf: |
    user nginx;
    worker_processes auto;
    error_log /var/log/nginx/error.log warn;
    pid /var/run/nginx.pid;

    events {
        worker_connections 4096;
        use epoll;
        multi_accept on;
    }

    http {
        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';

        access_log /var/log/nginx/access.log main;

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        client_max_body_size 100M;

        # Gzip 压缩
        gzip on;
        gzip_vary on;
        gzip_proxied any;
        gzip_comp_level 6;
        gzip_types text/plain text/css text/xml application/json application/javascript application/xml;

        # 虚拟服务器配置
        server {
            listen 80;
            server_name _;

            location / {
                root /usr/share/nginx/html;
                index index.html index.htm;
            }

            location /healthz {
                access_log off;
                return 200 "healthy\n";
                add_header Content-Type text/plain;
            }

            location /ready {
                access_log off;
                return 200 "ready\n";
                add_header Content-Type text/plain;
            }

            location /metrics {
                stub_status on;
                access_log off;
                allow 10.0.0.0/8;
                allow 172.16.0.0/12;
                allow 192.168.0.0/16;
                deny all;
            }
        }
    }

6.3 Service 配置

# nginx-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: default
  labels:
    app: nginx
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9113"
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  selector:
    app: nginx
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP
    nodePort: 30080
  - name: https
    port: 443
    targetPort: 443
    protocol: TCP
    nodePort: 30443
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800

6.4 Ingress 配置

# nginx-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "60"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "60"
    cert-manager.io/cluster-issuer: "letsencrypt"
spec:
  tls:
  - hosts:
    - nginx.example.com
    secretName: nginx-tls
  rules:
  - host: nginx.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-service
            port:
              number: 80

---

7 应用部署与验证

7.1 部署流程

# 1. 创建 ConfigMap
kubectl apply -f nginx-configmap.yaml

# 2. 创建 Deployment
kubectl apply -f nginx-deployment.yaml

# 3. 创建 Service
kubectl apply -f nginx-service.yaml

# 4. 创建 Ingress
kubectl apply -f nginx-ingress.yaml

# 5. 查看部署状态
kubectl get deployments
kubectl get pods -l app=nginx -o wide
kubectl get services
kubectl get ingress

# 6. 查看 Pod 详情
kubectl describe deployment nginx-deployment
kubectl describe pod nginx-deployment-5d5c9d8f8c-abc12

# 7. 查看事件日志
kubectl get events --sort-by='.lastTimestamp'

7.2 功能验证

# 1. 验证 Pod 间通信
kubectl run test-pod --image=busybox:1.28 --rm -it --restart=Never -- \
  wget -qO- http://nginx-service.default.svc.cluster.local

# 2. 验证外部访问
curl -v http://192.168.1.200:30080
curl -v https://nginx.example.com

# 3. 验证负载均衡
for i in {1..10}; do
    curl -s http://192.168.1.200:30080 | grep -o "Server Address: [^<]*"
done

# 4. 验证健康检查
kubectl get pods -l app=nginx -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}'

# 5. 验证自动扩缩容
kubectl autoscale deployment nginx-deployment --min=3 --max=10 --cpu-percent=80
kubectl get hpa

# 6. 压力测试
ab -n 10000 -c 100 http://192.168.1.200:30080/

# 输出示例:
# Server Software:        nginx/1.21.6
# Server Hostname:        192.168.1.200
# Server Port:            30080
# 
# Concurrency Level:      100
# Time taken for tests:   2.345 seconds
# Complete requests:      10000
# Failed requests:        0
# Requests per second:    4264.39 [#/sec]
# Time per request:       23.450 [ms]
# Transfer rate:          12345.67 [Kbytes/sec]

7.3 高可用验证

# 1. 删除 Pod 验证自动恢复
kubectl delete pod nginx-deployment-5d5c9d8f8c-abc12
watch kubectl get pods -l app=nginx

# 2. 删除节点验证 Pod 迁移
kubectl drain k8s-w-01 --ignore-daemonsets --delete-emptydir-data
watch kubectl get pods -o wide

# 3. 模拟网络分区
ssh ubuntu@k8s-w-01 "iptables -A INPUT -p tcp --dport 6443 -j DROP"
watch kubectl get nodes

# 恢复
ssh ubuntu@k8s-w-01 "iptables -D INPUT -p tcp --dport 6443 -j DROP"

# 4. 验证 Service 高可用
while true; do
    curl -s -o /dev/null -w "%{http_code}\n" http://nginx-service.default.svc.cluster.local
    sleep 1
done

---

8 总结与最佳实践

8.1 etcd 运维最佳实践

1. 硬件选型：使用 NVMe SSD，保证 IOPS > 5000
2. 参数调优：根据集群规模调整 election-timeout、heartbeat-interval
3. 定期备份：每 12 小时自动备份，保留 6 个快照
4. 监控告警：重点关注磁盘延迟、Leader 变更、提案失败
5. 容量规划：数据库使用率 < 80%，定期压缩

8.2 应用部署最佳实践

1. 资源限制：设置 requests 和 limits
2. 健康检查：配置 liveness 和 readiness probe
3. 亲和性：使用 podAntiAffinity 分散部署
4. 自动扩缩容：配置 HPA 应对流量波动
5. 滚动更新：使用 RollingUpdate 策略，maxUnavailable=0

8.3 性能优化建议

etcd 性能：

• 使用 NVMe SSD
• 调整 quota-backend-bytes=8GB
• 启用 auto-compaction-retention=8

应用性能：

• 使用本地镜像仓库
• 启用 Gzip 压缩
• 配置合理的缓存策略

---

参考文献：（略）

版权声明：本文版权归作者所有，欢迎转载，但必须保留此版权信息，且在文章开头明确标注原文链接。

---

本文是《使用 RKE 构建企业生产级 Kubernetes 集群》系列文章的第七篇（最终篇），至此完成了从架构设计、环境准备、集群部署、运维管理到应用验证的完整技术体系，感谢阅读！